{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. \n",
    "利用LightGBM/XGboost实现Happy Customer Bank目标客户（贷款成功的客户）识别\n",
    "1)\t文件说明\n",
    "Train.csv：训练数据\n",
    "Test.csv：测试数据\n",
    "2)\t字段说明\n",
    "数据集共26个字段: 其中1-24列为输入特征，25-26列为输出特征。\n",
    "    1. ID - 唯一ID（不能用于预测）\n",
    "    2. Gender - 性别\n",
    "    3. City - 城市\n",
    "    4. Monthly_Income - 月收入（以卢比为单位）\n",
    "    5. DOB - 出生日期\n",
    "    6. Lead_Creation_Date - 潜在（贷款）创建日期\n",
    "    7. Loan_Amount_Applied - 贷款申请请求金额（印度卢比，INR）\n",
    "    8. Loan_Tenure_Applied - 贷款申请期限（单位为年）\n",
    "    9. Existing_EMI -现有贷款的EMI（EMI：电子货币机构许可证） \n",
    "    10. Employer_Name雇主名称\n",
    "    11. Salary_Account - 薪资帐户银行\n",
    "    12. Mobile_Verified - 是否移动验证（Y / N）\n",
    "    13. VAR5 - 连续型变量\n",
    "    14. VAR1-  类别型变量\n",
    "    15. Loan_Amount_Submitted - 提交的贷款金额（在看到资格后修改和选择）\n",
    "    16. Loan_Tenure_Submitted - 提交的贷款期限（单位为年，在看到资格后修改和选择）\n",
    "    17. Interest_Rate - 提交贷款金额的利率\n",
    "1. 适当的特征工程（20分）\n",
    "2. 用LightGBM完成任务，并用交叉验证对模型的超参数（learning_rate、n_estimators、num_leaves、max_depth、min_data_in_leaf、colsample_bytree、subsample）进行调优。（70分）\n",
    "或者用XGBoost完成任务，并用交叉验证对模型的超参数（learning_rate、n_estimators、max_depth、min_child_weight、colsample_bytree、subsample、reg_lambda、reg_）进行调优。\n",
    "3. 对最终模型给出特征重要性（10分） "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "import lightgbm as lgbm\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = pd.read_csv(\"FE_train.csv\")\n",
    "y_train = train['Disbursed']\n",
    "x_train = train.drop(['ID','Disbursed'],axis=1)\n",
    "#特征名保留备用\n",
    "feat_names = x_train.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 超参数调优\n",
    "超参数说明：1树的数目n_estimator   ,learningrate\n",
    "2 max_depth ,最大叶子节点数目 num_leaves,（LightGBM采用叶子优先的方式生成树，num_leaves很重要，设置成比\n",
    "                                         2^max_depth 小）\n",
    "3 叶子结点最小样本数 min_data_in_leaf(min_data  ,min_child_samples)\n",
    "4 每棵树的lie采样比例：feature_fraction,colsample_bytree\n",
    "5 每棵树的行采样比例  bagging_fraction(同时要设置bagging_freq=1)  /subsample\n",
    "6 正则化参数 lambda_l1 (reg_alpha) lambda_l2 (reg_lambda)\n",
    "7 (1)特征最大bin数max_bin默认255，2) (2)建立直方图的样本数目subsample_for_bin默认200000\n",
    "n_estimator 使用lightgbm内嵌CV调优  其他参数用GridSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "MAX_ROUNDS = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import StratifiedKFold\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3 )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# # 1.n_estimators调优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_n_estimators(params, x_train, y_train, early_stopping_rounds=10 ):#num_boost_round为弱分类器数目,下面的代码参数里因为已经设置了early_stopping_rounds#即性能未提升的次数超过过早停止设置的数值，则停止训练\n",
    "    lgbm_params = params.copy()\n",
    "    lgbmtrain = lgbm.Dataset(x_train, y_train)\n",
    "    \n",
    "    cv_result = lgbm.cv(lgbm_params , lgbmtrain , num_boost_round=MAX_ROUNDS , nfold=5,  metrics='auc' , early_stopping_rounds=early_stopping_rounds,seed=3 )\n",
    "     \n",
    "    print('best n_estimators:' , len(cv_result['auc-mean']))\n",
    "    print('best cv score:' , cv_result['auc-mean'][-1])\n",
    "     \n",
    "    return len(cv_result['auc-mean'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 68\n",
      "best cv score: 0.8374021335350141\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type':'goss',\n",
    "          'objective':'binary',\n",
    "          'is_unbalance':True,\n",
    "          #'categorical_feature': names:'City', 'Employer_Name', 'Salary_Account','Device_Type','Filled_Form','Gender','Mobile_Verified','Source','Var1','Var2','Var4',\n",
    "          'Categorical_feature':[0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_job':4,\n",
    "          'learnning_rate':0.1,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'num_leaves':60, # 预先设置为一般取值 \n",
    "          'max_depth':6,  # 预先设置为一般取值 \n",
    "          'colsample_bytree':0.7, # 预先设置为一般取值 \n",
    "          'verbosity':5\n",
    "         }\n",
    "n_estimators_1 = get_n_estimators(params, x_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# # 2.num_leaves  and  max_depth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 4 candidates, totalling 20 fits\n",
      "[CV] num_leaves=50 ...................................................\n",
      "[CV] num_leaves=50 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] num_leaves=50 ...................................................\n",
      "[CV] num_leaves=50 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=50, score=0.8136223632310067, total=   4.4s\n",
      "[CV] num_leaves=50 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=50, score=0.8103194721237961, total=   4.5s\n",
      "[CV] num_leaves=60 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=50, score=0.8256559116185467, total=   5.7s\n",
      "[CV] num_leaves=60 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=50, score=0.8493103527125136, total=   6.5s\n",
      "[CV] num_leaves=60 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=50, score=0.8275227131537708, total=   5.3s\n",
      "[CV] num_leaves=60 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=60, score=0.8534475504487509, total=   6.3s\n",
      "[CV] num_leaves=60 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=60, score=0.8090632824558395, total=   5.5s\n",
      "[CV] num_leaves=70 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=60, score=0.8070129282105285, total=   6.0s\n",
      "[CV] num_leaves=70 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=60, score=0.8242401131720452, total=   5.5s\n",
      "[CV] num_leaves=70 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=60, score=0.8290402140020561, total=   6.3s\n",
      "[CV] num_leaves=70 ...................................................\n",
      "[CV] .......... num_leaves=70, score=0.8522101411993369, total=   6.1s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  10 tasks      | elapsed:   20.7s\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] num_leaves=70 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=70, score=0.8101793860401303, total=   6.2s\n",
      "[CV] num_leaves=80 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=70, score=0.8061746011600746, total=   5.6s\n",
      "[CV] num_leaves=80 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=70, score=0.8292867791928364, total=   5.5s\n",
      "[CV] num_leaves=80 ...................................................\n",
      "[CV] .......... num_leaves=70, score=0.8244684270288711, total=   5.7s\n",
      "[CV] num_leaves=80 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=80, score=0.8522101411993369, total=   4.9s\n",
      "[CV] num_leaves=80 ...................................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=80, score=0.8101793860401303, total=   5.8s\n",
      "[CV] .......... num_leaves=80, score=0.8061746011600746, total=   4.5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  18 out of  20 | elapsed:   32.9s remaining:    3.7s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .......... num_leaves=80, score=0.8244684270288711, total=   4.9s\n",
      "[CV] .......... num_leaves=80, score=0.8292867791928364, total=   4.8s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  20 out of  20 | elapsed:   33.3s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss',\n",
       "        categorical_feature=[0, 1, 3, 5, 6, 12, 15, 16, 17, 18, 19, 20],\n",
       "        class_weight=None, colsample_bytree=0.7, importance_type='split',\n",
       "        is_unbalance=True, learning_rate=0.1, max_depth=6,\n",
       "        min_child_samples=20, min_child_weight=....0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'num_leaves': range(50, 90, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=5)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':68,          \n",
    "          #'num_leaves': 60,\n",
    "          'max_depth': 6,\n",
    "          'colsample_bytree': 0.7, \n",
    "         }\n",
    "lg = LGBMClassifier(silent=False, **params)\n",
    "\n",
    "num_leaves_s = range(50,90,10)\n",
    "tuned_parameters = dict( num_leaves = num_leaves_s)\n",
    "grid_search = GridSearchCV(lg, n_jobs=4, param_grid=tuned_parameters, cv=kfold, scoring=\"roc_auc\", verbose=5, refit = False)\n",
    "\n",
    "grid_search.fit(x_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8252862746581494\n",
      "{'num_leaves': 50}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAELCAYAAADkyZC4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VfWd//HXJ3vYQoBAgUDCDolUlAzuVUFFEXB+2k7LSNXWukyrM6NOH2rrErdp3TrTcdSptmrrr9WqoxYQxQ2tUrUmBdSERYSwCoR9JyR85o97gjEGCJCTc2/u+/l45JF7T8499/P1mLw533vO55i7IyIi0tJSoi5ARETaJgWMiIiEQgEjIiKhUMCIiEgoFDAiIhIKBYyIiIRCASMiIqFQwIiISCgUMCIiEoq0qAuIUrdu3bywsDDqMkREEkp5efk6d8872HpJHTCFhYWUlZVFXYaISEIxs6XNWU9TZCIiEgoFjIiIhEIBIyIioVDAiIhIKBQwIiISCgWMiIiEQgEjIiKhUMAchlWbdvLYu0uiLkNEJK4pYA7DC7NXcvu0Sv7y2bqoSxERiVsKmMNw6cn96NMlm9umVFJbtzfqckRE4pIC5jBkpafy03FFLFizlT/8dVnU5YiIxCUFzGEaW9yDEwd05f5XF7Jxe03U5YiIxB0FzGEyM26dUMy23bX84rWFUZcjIhJ3FDBHYMjXOjL5uL78/oOlzPt8S9TliIjEFQXMEbrmzMHkZKdz29QK3D3qckRE4oYC5gh1bpfBtWcN4f3FG3j5k9VRlyMiEjcUMC3gH0f1ZejXOnLXS/PYtacu6nJEROKCAqYFpKYYpROLWblpJ796e3HU5YiIxAUFTAs5vn9Xzh3ek4ffXsSqTTujLkdEJHIKmBZ047ihuMPPXp4fdSkiIpFTwLSg/Nx2XHnqAKbOXcVfl2yIuhwRkUgpYFrYlacOoFdOFqVTKqjbq9OWRSR5KWBaWHZGKj85dxiVn2/hjx8uj7ocEZHIKGBCcO7wnhzXrwv3vbqAzTv2RF2OiEgkFDAhqO9TtmlHDf/5hvqUiUhyCjVgzOxsM1tgZovM7IYmft7XzGaa2Wwz+8jMxgXLzzSzcjP7OPg+usFr3gq2OSf46h4sv9bMKoPtvGFmBWGO7WCKenVi0qi+/O69pXy6ZmuUpYiIRCK0gDGzVOBB4BygCJhkZkWNVrsJeMbdjwG+AzwULF8HTHD34cDFwJONXnehu48IvtYGy2YDJe7+deA54J4WH9Qhuu6sIbTPSOX2aZXqUyYiSSfMI5hRwCJ3X+zuNcDTwHmN1nGgU/A4B1gF4O6z3X1VsLwCyDKzzAO9mbvPdPcdwdP3gfwWGMMR6dI+g2vPHMw7n67jtco1UZcjItKqwgyY3kDD06hWBMsaKgUmm9kKYDpwdRPbuQCY7e67Gyx7PJgeu9nMrInXXAq8fNiVt6DJxxcwuEcH7lSfMhFJMmEGTFN/+BvPE00CnnD3fGAc8KSZ7avJzIqBu4ErGrzmwmDq7JTg67tfelOzyUAJcG+TRZldbmZlZlZWXV19iEM6dGmpKdw6oZhlG3bwm3eXhP5+IiLxIsyAWQH0afA8n2AKrIFLgWcA3P09IAvoBmBm+cALwEXu/ln9C9x9ZfB9K/AHYlNxBK85A/gpMLHREQ8NXv+Iu5e4e0leXt4RDbC5ThrYjbHFPXhw5iJWb97VKu8pIhK1MAPmQ2CQmfUzswxiH+JPabTOMmAMgJkNIxYw1WbWGXgJuNHdZ9WvbGZpZlYfQOnAeOCT4PkxwK+Ihcta4sxPxxVRu9e5+xX1KROR5BBawLh7LXAVMAOYR+xssQozu93MJgarXQdcZmZzgaeASzx2utVVwEDg5kanI2cCM8zsI2AOsBJ4NNjWvUAH4Nlg/cZhFqm+Xdtx2Sn9eGH2SsqXboy6HBGR0Fkynz5bUlLiZWVlrfZ+23fXMvr+t+jRKYsXf3gSKSlNfUwlIhLfzKzc3UsOtp6u5G9F7TPTuPGcYXy0YjPPla+IuhwRkVApYFrZeSN6MbIgl3tmzGfLLvUpE5G2SwHTysyM0gnFrN9ewwNvfBp1OSIioVHARGB4fg7/MLIPj8+q4rPqbVGXIyISCgVMRH589hCy01O5Y1pl1KWIiIRCARORbh0y+ZczBvHWgmrenK8+ZSLS9ihgInTRCYX0z2vPHdPmUVO7N+pyRERalAImQhlpKdwyvogl67bz+Cz1KRORtkUBE7HThnRnzNDuPPDmItZuVZ8yEWk7FDBx4KbxReyureOeVxZEXYqISItRwMSBft3a8/2T+/Fc+QrmLt8UdTkiIi1CARMnrh49iLyOmZROrWDv3uTtDycibYcCJk50yEzj+rOHMnvZJl6cszLqckREjpgCJo6cf0xvju7TmZ+/PJ9tu2ujLkdE5IgoYOJISopROqGItVt38+DMRVGXIyJyRBQwceaYvrlccGw+v3lnCVXrtkddjojIYVPAxKHrzx5Ceqpx50vzoi5FROSwKWDiUPdOWVw1ehCvz1vDnxdWR12OiMhhUcDEqe+fXEhh13bcPq2SPXXqUyYiiUcBE6cy01K56dwiFq3dxu/eWxp1OSIih0wBE8fGDOvONwbn8Z+vL2T9tt1RlyMickgUMHHMzLhlfBE7a+q471X1KRORxKKAiXMDu3fg4hMLefrD5XyycnPU5YiINJsCJgH885hBdGmXQemUCtzVp0xEEoMCJgHkZKfz47FDKFu6kSlzV0VdjohIsyhgEsS3SvpwVO9O/Gz6fHbUqE+ZiMQ/BUyCSE0xSicUs3rLLh5+67OoyxEROSgFTAIpKezCeSN68as/L2b5hh1RlyMickAKmARzwzlDSTXjLvUpE5E4p4BJMD1zsvnR6QN4pWI1f1m0LupyRET2SwGTgH5wSn/6dMnmtqmV1KpPmYjEKQVMAspKT+Wn44pYsGYrv/9gWdTliIg0SQGToMYW9+CkgV35xWsL2bi9JupyRES+QgGToMyMWycUs213Lfe/pj5lIhJ/FDAJbHCPjnz3+AL+8MEyKldtibocEZEvUcAkuGvOGExOdjq3T1OfMhGJLwqYBJfTLp3rzhrC+4s38PInq6MuR0RkHwVMGzBpVF+Gfq0jd700j1176qIuR0QEUMC0CakpRunEYlZu2smv3l4cdTkiIkDIAWNmZ5vZAjNbZGY3NPHzvmY208xmm9lHZjYuWH6mmZWb2cfB99ENXvNWsM05wVf3YHmmmf0xeK8PzKwwzLHFm+P7d+Xc4T15+O1FrNy0M+pyRETCCxgzSwUeBM4BioBJZlbUaLWbgGfc/RjgO8BDwfJ1wAR3Hw5cDDzZ6HUXuvuI4GttsOxSYKO7DwT+A7i7xQcV524cNxR3+Nl09SkTkeiFeQQzCljk7ovdvQZ4Gjiv0ToOdAoe5wCrANx9trvX31mrAsgys8yDvN95wG+Dx88BY8zMjnAMCSU/tx1XnjqAaR99zgeL10ddjogkuTADpjewvMHzFcGyhkqByWa2ApgOXN3Edi4AZrv77gbLHg+mx25uECL73s/da4HNQNcjHkWCufLUAfTKyaJ0aiV1e3XasohEJ8yAaeroofFfvEnAE+6eD4wDnjSzfTWZWTGxqa4rGrzmwmDq7JTg67uH8H6Y2eVmVmZmZdXV1c0eTKLIzkjlJ+cOY97nW3j6Q/UpE5HohBkwK4A+DZ7nE0yBNXAp8AyAu78HZAHdAMwsH3gBuMjd993C0d1XBt+3An8gNhX3pfczszRiU24bGhfl7o+4e4m7l+Tl5R3hEOPTucN7cly/Ltw3YwGbd+yJuhwRSVJhBsyHwCAz62dmGcQ+xJ/SaJ1lwBgAMxtGLGCqzawz8BJwo7vPql/ZzNLMrD6A0oHxwCfBj6cQOyEA4JvAm56kl7bX9ynbvHMP//H6wqjLEZEkFVrABJ+DXAXMAOYRO1uswsxuN7OJwWrXAZeZ2VzgKeCSIBSuAgYCNzc6HTkTmGFmHwFzgJXAo8G2fgN0NbNFwLXAV06LTiZFvToxaVRfnnx/KQvXbI26HBFJQpak/8gHoKSkxMvKyqIuIzQbttdw2r0z+Xp+Z568dBRJdlKdiITEzMrdveRg6+lK/jasS/sMrj1zMO8uWserlWuiLkdEkowCpo2bfHwBg3t04M6XKtWnTERalQKmjUtLTeHWCcUs37CT37y7JOpyRCSJKGCSwEkDuzG2uAcPzlzE6s27oi5HRJKEAiZJ3HRuEbV7nZ+/rD5lItI6FDBJok+Xdlx+Sn9enLOK8qVfuf5URKTFKWCSyA9PH8DXOmVROqWSvepTJiIhU8AkkXYZadw4bigfr9zMs+XLD/4CEZEjoIBJMhOP7kVJQS73zljAll3qUyYi4VHAJJn6PmXrt9fwX69/GnU5ItKGKWCS0PD8HP5hZB+e+EsVi9Zui7ocEWmjFDBJ6sdnDyE7PZU7plWSzP3oRCQ8Cpgk1a1DJv9yxiDeXljNm/PXRl2OiLRBCpgkdtEJhfTPa88d0yqpqd0bdTki0sYoYJJYRloKt4wvomr9Dh6fpT5lItKyFDBJ7rQh3RkztDsPvLmItVvVp0xEWo4CRrhpfBG7a+u455UFUZciIm3IfgPGzMaa2TebWH6hmZ0ZblnSmvp1a8/3T+7Hc+UrmLN8U9TliEgbcaAjmNuAt5tY/gZwezjlSFSuHj2IvI6ZlE6pUJ8yEWkRBwqYdu5e3Xihu68G2odXkkShQ2Ya1589lDnLN/HC7JVRlyMibcCBAibLzNIaLzSzdCA7vJIkKucf05uj+3Tm56/MZ9vu2qjLEZEEd6CAeR541Mz2Ha0Ej/8n+Jm0MSkpRumEIqq37ua/31wUdTkikuAOFDA3AWuApWZWbmZ/A6qA6uBn0gYd0zeXC47N57F3l1C1bnvU5YhIAttvwLh7rbvfAPQBLgEuBvq6+w3urj7vbdj1Zw8hPdW486XKqEsRkQR2oNOUzzez84FzgEHAQKDEzDq2VnESje6dsrh6zCBen7eWtxd+5TwPEZFmOdAU2YRGXxOBfwM+MrPRrVCbROh7JxVS2LUdt0+tYE+d+pSJyKH7ylli9dz9e00tN7MC4BnguLCKkuhlpqVy8/giLv1tGb/9SxU/OKV/1CWJSII55FYx7r4USA+hFokzo4d259TBefzy9U9Zt2131OWISII55IAxs6GA/tokATPj5vFF7NxTx30z1KdMRA7NfqfIzGwq0LhnSBegJzA5zKIkfgzs3oGLTyzksVlLuPC4Aobn50RdkogkiP0GDHBfo+cObCAWMpOB98IqSuLLP48ZxIuzV3Lb1AqevfIEzCzqkkQkARzoOpi367+AzcB4YBqxJpjzWqk+iQM52en8eOwQypZuZMrcVVGXIyIJ4kDXwQw2s1vMbB7w38BywNz9dHf/71arUOLCt0r6cFTvTvxs+nx21KhPmYgc3IE+5J8PjAEmuPvJ7v4AUNc6ZUm8SU0xSicUs3rLLh6a+VnU5YhIAjhQwFwArAZmmtmjZjYG0OR7Eisp7MJ5I3rxyDuLWbZ+R9TliEicO9BnMC+4+7eBocBbwDVADzN72MzOaqX6JM7ccM5QUs24a7r6lInIgR30Ohh33+7uv3f38UA+MAe4IfTKJC71zMnmR6cPYEbFGmYtWhd1OSISxw7pQkt33+Duv3J39SJLYj84pT99umRz29QKatWnTET245Cv5BfJSk/lp+OKWLhmG7//YFnU5YhInAo1YMzsbDNbYGaLzOwr02pm1tfMZprZbDP7yMzGBcvPDG5y9nHw/StHTGY2xcw+afB8hJm9b2ZzzKzMzEaFObZkN7a4BycN7MovXlvIxu01UZcjInEotIAxs1TgQWL3kykCJplZUaPVbgKecfdjgO8ADwXL1xE7PXo4sRudPdlo2+cD2xpt6x7gNncfAdwSPJeQmBm3Tihm2+5a7n9NfcpE5KvCPIIZBSxy98XuXgM8DZzXaB0HOgWPc4BVAO4+293rLxmvALLMLBPAzDoA1wJ3NmdbEp7BPTry3eML+MMHy6hctSXqckQkzoQZML2JXf1fb0WwrKFSYLKZrQCmA1c3sZ0LgNnuXt/B+Q7gfqDxhRj/CtxrZsuJ9VG78Yiql2a55ozB5GSnc9vUCtwb90YVkWQWZsA0dVFm479Ak4An3D0fGAc8aWb7ajKzYuBu4Irg+QhgoLu/0MS2/wm4xt37ELtm5zdNFmV2efAZTVl1tW4HfKRy2qVz3VlD+GDJBqZ/vDrqckQkjoQZMCuAPg2e5/PVaatLid0dE3d/D8gCugGYWT7wAnCRu9f3JjkBGGlmVcC7wGAzeyv42cXA88HjZ4lN0X2Fuz/i7iXuXpKXl3fYg5MvTBrVl2E9O/Hv0+exs0bdhEQkJsyA+RAYZGb9zCyD2If4Uxqts4xYvzPMbBixgKk2s87AS8CN7j6rfmV3f9jde7l7IXAysNDdTwt+vAo4NXg8Gvg0lFHJV8T6lBWxctNOfvVn9SkTkZjQAsbda4GrgBnE2vs/4+4VZna7mU0MVrsOuMzM5gJPAZd4bCL/KmAgcHNw2vEcM+t+kLe8DLg/2Na/A5eHMCzZj+P6d+Xcr/fkf97+jJWbdkZdjojEAUvmD2ZLSkq8rKws6jLajJWbdjLm/rcYM6wHD/7jsVGXIyIhMbNydy852Hq6kl9aTO/O2Vx56gBe+uhz3l+8PupyRCRiChhpUVd8YwC9crK4bWoldXuT9+hYRBQw0sKyM1L5ybnDmPf5Fp76q/qUiSQzBYy0uHOH9+S4fl24/9UFbN6xJ+pyRCQiChhpcfV9yjbv3MN/vL4w6nJEJCIKGAlFUa9OTBrVlyffX8qC1VujLkdEIqCAkdBcd9YQ2mekcvs09SkTSUYKGAlNl/YZXHvmYGYtWs+MijVRlyMirUwBI6GafHwBg3t04K7plezaoz5lIslEASOhSktN4dYJxSzfsJNfv7M46nJEpBUpYCR0Jw3sxtjiHjw48zM+36w+ZSLJQgEjreKmc4uoc+fnL8+PuhQRaSUKGGkVfbq04/JT+vOnOasoq9oQdTki0goUMNJqfnj6AL7WKYvSqRXqUyaSBBQw0mraZaRx47ihfLJyC8+WLY+6HBEJmQJGWtXEo3tRUpDLvTMWsHmn+pSJtGUKGGlVZkbpxGI27KjhgTd0V2uRtkwBI63uqN45fLukD0/8pYpFa7dFXY6IhEQBI5H4t7FDyM5I5Y5plepTJtJGKWAkEt06ZPIvYwbx9sJq3py/NupyRCQEChiJzMUnFjIgrz13TKtkd636lIm0NQoYiUx6ago3jy+iav0OHp9VFXU5ItLCFDASqdOGdGfM0O488ManrN2yK+pyRKQFKWAkcjeNL6Kmbi93v7Ig6lJEpAUpYCRy/bq15/sn9+N//7aC2cs2Rl2OiLQQBYzEhatHDyKvYyalUyvZqz5lIm2CAkbiQofMNK4/eyhzl2/i+dkroy5HRFqAAkbixvnH9OboPp25+5X5bNtdG3U5InKEFDASN1JSjNIJRVRv3c0Db6pPmUiiU8BIXDmmby4XHJvPY+8uYcm67VGXIyJHQAEjcef6s4eQkZrCndMqoy5FRI6AAkbiTvdOWVw9ZhBvzF/LWwvUp0wkUSlgJC5976RCCru24/ZpldTU7o26HBE5DAoYiUuZaancPL6IxdXb+d17VVGXIyKHQQEjcWv00O6cOjiPX77+KdVbd0ddjogcIgWMxC0z4+bxRezcU8d9M9SnTCTRKGAkrg3s3oFLTizkmfLlfLxic9TliMghUMBI3PvnMwbRtX0GpVMrdHtlkQSigJG41ykrnR+PHUL50o38ac6qqMsRkWZSwEhC+NbIPgzvncPPXp7HdvUpE0kIoQaMmZ1tZgvMbJGZ3dDEz/ua2Uwzm21mH5nZuGD5mWZWbmYfB99HN/HaKWb2SaNlVwfvV2Fm94Q3MmltKSlG6cQi1mzZzUNvLYq6HBFphtACxsxSgQeBc4AiYJKZFTVa7SbgGXc/BvgO8FCwfB0wwd2HAxcDTzba9vnAtkbLTgfOA77u7sXAfS07IonayIIunDeiF4++s4Rl63dEXY6IHESYRzCjgEXuvtjda4CniQVAQw50Ch7nAKsA3H22u9dPtlcAWWaWCWBmHYBrgTsbbeufgJ+7++5gG+ox0gbdcM5QUs24a7r6lInEuzADpjewvMHzFcGyhkqByWa2ApgOXN3Edi4AZtcHB3AHcD/Q+J+wg4FTzOwDM3vbzP6uqaLM7HIzKzOzsurq6kMakESvZ042Pzp9ADMq1jBr0bqoyxGRAwgzYKyJZY3PMZ0EPOHu+cA44Ekz21eTmRUDdwNXBM9HAAPd/YUmtp0G5ALHAz8GnjGzr9Tg7o+4e4m7l+Tl5R3GsCRqPzilP326ZHPb1Apq69SnTCRehRkwK4A+DZ7nE0yBNXAp8AyAu78HZAHdAMwsH3gBuMjdPwvWPwEYaWZVwLvAYDN7q8H7Pe8xfwX21m9L2pas9FR+Oq6IhWu28f/fXxp1OSKyH2EGzIfAIDPrZ2YZxD7En9JonWXAGAAzG0YsYKrNrDPwEnCju8+qX9ndH3b3Xu5eCJwMLHT304IfvwiMDrY1GMggdrKAtEFji3tw0sCu/OK1hWzYXhN1OSLShNACxt1rgauAGcA8YmeLVZjZ7WY2MVjtOuAyM5sLPAVc4rFLta8CBgI3m9mc4Kv7Qd7yMaB/cOry08DFrsu+2ywz49YJxWyvqeP+V9WnTCQeWTL/DS4pKfGysrKoy5AjUDqlgt+9V8W0q0+hqFeng64vIkfOzMrdveRg6+lKfklo15wxmJzsdPUpE4lDChhJaDnt0rnurCH8dckGXvr486jLEZEGFDCS8CaN6suwnp3495fmsbOmLupyRCSggJGEl5pilE4oYtXmXXzj3pn88Pfl/PqdxcxZvok9uk5GJDJpURcg0hKO69+Vhy48lhkVqymr2sj0j1cDkJWewtH5nSkpzKWkoAvH9s0lp116xNWKJAedRaazyNqk1Zt3UbZ0A+VLN1K+dCMVq7ZQtzf2//qg7h0oKcxlZEEXSgpyKejajiaaPojIfjT3LDIFjAImKeyoqWXO8k2UV22kbOlG/rZsI1t3xe4r061DBsf2zd0XOkf17kRmWmrEFYvEr+YGjKbIJCm0y0jjxAHdOHFArHvQ3r3Op2u3xY5ygtB5tXINABlpKRydn8OxBbFptZEFuXRpnxFl+SIJSUcwOoKRwNqtu/jb0o2UBYFTsWoze+pivx/989pTUpDLyILYUc6AvPaaVpOkpSmyZlDAyIHs2lPH3OWbKF+2kfKqjZQv28imHXsAyG2Xvi9sSgpzGd47h6x0TatJctAUmcgRykpP5bj+XTmuf1cgNq22eN02yqo27jt54PV5sfvapacaR/XOCY5yYtNqeR0zoyxfJHI6gtERjByB9dt27wub8qUb+WjFZmqCa28Ku7bb9zlOSWEuA/M6kJKiaTVJfJoiawYFjLS03bV1fLJy877Pcf62dCPrg9sJ5GSnc2zfzpQUxq7HGdGnM9kZmlaTxKMpMpEIZKalBlNkXbgCcHeq1u+grCp2TU7Z0o3MXBC7vUBailHcq9O+z3FGFuTSo1NWtAMQaUE6gtERjLSyjdtrmL38i7PV5i7fxO7a2LRafm527HOcwthFoIN7dCRV02oSZ3QEIxKncttnMHpoD0YP7QFATe1eKlZt3vc5zqzP1vPinNjdxTtmpjGib+d9n+OM6NOZ9pn6tZXEoCMYHcFInHF3lm/YSdnSDfs+x1mwZivuscaew3p2ZGTfL45yenXOjrpkSTL6kL8ZFDCSKDbv3MPsZbEjnLKqjcxZvomde2K3JuiVk7UvbEYW5DL0ax1JS1WjdAmPpshE2pCc7HROG9Kd04Z0B6C2bi/zPt+67yjnwyUbmDo3Nq3WPiOVEX0772vmOaJvZzplqYO0tD4dwegIRtoAd2fV5l1fnK1WtZH5q7ew18EMhvTouO+WBSMLcsnPzVarGzlsmiJrBgWMtGVbd+2JdZAOTh6YvWwT23bHOkj36JT5RaubglyKenUiXdNq0kyaIhNJch2z0jllUB6nDMoDoG6vM3/1ln2B0/DGbNnpqRzdJ4eRBboxm7QcHcHoCEaSWP2N2er7q1V+/sWN2Qb36LDvCGekbswmDWiKrBkUMCJfduAbs2UysiB2Tc7IwlyKe+nGbMlKU2Qicsiac2O2GRVfvjFbw6OcXN2YTRrQEYyOYEQOSXNuzFZ/lNO/m27M1hZpiqwZFDAiR67+xmz1XQca3pitS/sMBnXvoH5qcej7J/XjjKIeh/VaTZGJSKs42I3ZqtZvZ29d8v5DNl7VtcLBhQJGRFpUSooxsHtHBnbvyHdG9Y26HImQrqwSEZFQKGBERCQUChgREQmFAkZEREKhgBERkVAoYEREJBQKGBERCYUCRkREQpHUrWLMrBpYepgv7wasa8FyoqSxxJ+2Mg7QWOLVkYylwN3zDrZSUgfMkTCzsub04kkEGkv8aSvjAI0lXrXGWDRFJiIioVDAiIhIKBQwh++RqAtoQRpL/Gkr4wCNJV6FPhZ9BiMiIqHQEYyIiIRCAdNMZlZlZh+b2RwzKwuWdTGz18zs0+B7btR1Hsx+xlFqZiuDZXPMbFzUdTaHmXU2s+fMbL6ZzTOzExJxn8B+x5Jw+8XMhjSod46ZbTGzf020/XKAcSTcPgEws2vMrMLMPjGzp8wsy8z6mdkHwT75o5lltPj7aoqsecysCihx93UNlt0DbHD3n5vZDUCuu18fVY3NsZ9xlALb3P2+qOo6HGb2W+Add/918MvRDvgJCbZPYL9j+VcScL/UM7NUYCVwHPAjEnC/wFfG8T0SbJ+YWW/gXaDI3Xea2TPAdGAc8Ly7P21m/wPMdfeHW/K9dQRzZM4Dfhs8/i3w9xHWklTMrBPwDeA3AO5e4+6bSMB9coCxJLoxwGfuvpQE3C8NNBxHokoDss0sjdg/Xj4HRgPPBT8PZZ8oYJrPgVfNrNzMLg+W9XD3zwGC790jq675mhoHwFVm9pGZPRbv0xeB/kA18LiZzTazX5tZexJzn+xvLJB4+6Wh7wBPBY8Y02XrAAAEcUlEQVQTcb/UazgOSLB94u4rgfuAZcSCZTNQDmxy99pgtRVA75Z+bwVM853k7scC5wA/MrNvRF3QYWpqHA8DA4ARxP4HvD/C+porDTgWeNjdjwG2AzdEW9Jh299YEnG/ABBM800Eno26liPRxDgSbp8EIXge0A/oBbQn9vvfWIt/XqKAaSZ3XxV8Xwu8AIwC1phZT4Dg+9roKmyepsbh7mvcvc7d9wKPEhtbvFsBrHD3D4LnzxH7I51w+4T9jCVB90u9c4C/ufua4Hki7hdoNI4E3SdnAEvcvdrd9wDPAycCnYMpM4B8YFVLv7ECphnMrL2Zdax/DJwFfAJMAS4OVrsY+FM0FTbP/sZR/4sf+H/ExhbX3H01sNzMhgSLxgCVJNg+gf2PJRH3SwOT+PK0UsLtl8CXxpGg+2QZcLyZtTMz44vflZnAN4N1QtknOousGcysP7F/7UNsOuMP7n6XmXUFngH6EtuJ33L3DRGVeVAHGMeTxA75HagCrqifL49nZjYC+DWQASwmdoZPCgm0T+rtZyz/RWLul3bAcqC/u28OliXU7wrsdxyJ+rtyG/BtoBaYDfyA2GcuTwNdgmWT3X13i76vAkZERMKgKTIREQmFAkZEREKhgBERkVAoYEREJBQKGBERCYUCRkREQqGAEYkjZvaEmX3z4GuKxD8FjIiIhEIBI3IQZlYY3ATs0eCmTa+aWbaZvWVmJcE63YJ77WBml5jZi2Y21cyWmNlVZnZt0Cn5fTPr0sz3HWlmbwedr2c06OV1mZl9aGZzzex/gxYgORa7mVxKsE47M1tuZulmNsDMXgm2846ZDQ3W+VZwA6q5ZvbnUP7jSVJTwIg0zyDgQXcvBjYBFxxk/aOAfyTWDPEuYEfQKfk94KKDvZmZpQMPAN9095HAY8F2IHaTqL9z96OBecClQSuTucCpwToTgBlBc8NHgKuD7fwb8FCwzi3A2GA7Ew9Wk8ihSjv4KiJCrBvtnOBxOVB4kPVnuvtWYKuZbQamBss/Br7ejPcbQiykXov1JySVWHt4gKPM7E6gM9ABmBEs/yOxflMzid3D5CEz60Csc+6zwXYAMoPvs4AngjscPt+MmkQOiQJGpHkaNgGsA7KJNQ6snwXIOsD6exs830vzfu8MqHD3E5r42RPA37v7XDO7BDgtWD4F+FkwBTcSeJPYvT82ufuIxhtx9yvN7DjgXGCOmY1w9/XNqE2kWTRFJnL4qoj9IYcv2p63lAVAnpmdALEpMzMrDn7WEfg8mEa7sP4F7r4N+CvwS2BacN+SLcASM/tWsB0zs6ODxwPc/QN3vwVYB/Rp4TFIklPAiBy++4B/MrO/AN1acsPuXkMstO42s7nAHGJTXQA3Ax8ArwHzG730j8Dk4Hu9C4FLg+1UELu7IcC9ZvaxmX0C/JnYZzgiLUbt+kVEJBQ6ghERkVDoQ36RCJjZg8BJjRb/0t0fj6IekTBoikxEREKhKTIREQmFAkZEREKhgBERkVAoYEREJBQKGBERCcX/AW+Zmz2dfmBEAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "n_leafs = len(num_leaves_s)\n",
    "\n",
    "x_axis = num_leaves_s\n",
    "plt.plot(x_axis, test_means)\n",
    "#plt.errorbar(x_axis, -test_means, yerr=test_stds,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_means, yerr=train_stds,label = ' Train')\n",
    "plt.xlabel( 'num_leaves' )\n",
    "plt.ylabel( 'AUC' )\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.82528627, 0.82456092, 0.82446397, 0.82446397])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means   #性能抖动，取系统推荐值：70"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. min_child_samples\n",
    "\n",
    "叶子节点的最小样本数目\n",
    "\n",
    "叶子节点数目：70，共2类，平均每类35个叶子节点 每棵树的样本数目数目最少的类（稀有事件）的样本数目：8w * 4/5 * 1.4% = 840 所以每个叶子节点约840/35 = 25个样本点\n",
    "\n",
    "搜索范围：10-50\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 4 candidates, totalling 20 fits\n",
      "[CV] min_child_samples=10 ............................................\n",
      "[CV] min_child_samples=10 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] min_child_samples=10 ............................................\n",
      "[CV] min_child_samples=10 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .... min_child_samples=10, score=0.852096953066941, total=   4.3s\n",
      "[CV] min_child_samples=10 ............................................\n",
      "[CV] ... min_child_samples=10, score=0.8023864651114396, total=   4.2s\n",
      "[CV] min_child_samples=20 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=10, score=0.8108487966615219, total=   5.6s\n",
      "[CV] min_child_samples=20 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=10, score=0.8259728420150758, total=   6.2s\n",
      "[CV] min_child_samples=20 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=10, score=0.8259340435818898, total=   5.3s\n",
      "[CV] min_child_samples=20 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=20, score=0.8522101411993369, total=   5.7s\n",
      "[CV] min_child_samples=20 ............................................\n",
      "[CV] ... min_child_samples=20, score=0.8101793860401303, total=   4.6s\n",
      "[CV] min_child_samples=30 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=20, score=0.8061746011600746, total=   6.0s\n",
      "[CV] min_child_samples=30 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=20, score=0.8244684270288711, total=   5.8s\n",
      "[CV] min_child_samples=30 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=20, score=0.8292867791928364, total=   5.4s\n",
      "[CV] min_child_samples=30 ............................................\n",
      "[CV] ... min_child_samples=30, score=0.8530135482764534, total=   5.5s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  10 tasks      | elapsed:   18.9s\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] min_child_samples=30 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=30, score=0.8103634596695821, total=   5.5s\n",
      "[CV] min_child_samples=40 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=30, score=0.8069411238750559, total=   4.3s\n",
      "[CV] min_child_samples=40 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=30, score=0.8286507374227647, total=   5.2s\n",
      "[CV] min_child_samples=40 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=30, score=0.8278162267444718, total=   5.6s\n",
      "[CV] min_child_samples=40 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=40, score=0.8557378379923398, total=   5.3s\n",
      "[CV] min_child_samples=40 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=40, score=0.8117361230206368, total=   5.3s\n",
      "[CV] ... min_child_samples=40, score=0.8064860581820926, total=   5.6s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  18 out of  20 | elapsed:   31.5s remaining:    3.5s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=40, score=0.8294663080375202, total=   5.3s\n",
      "[CV] ... min_child_samples=40, score=0.8334817162957552, total=   4.0s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  20 out of  20 | elapsed:   32.1s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss',\n",
       "        categorical_feature=[0, 1, 3, 5, 6, 12, 15, 16, 17, 18, 19, 20],\n",
       "        class_weight=None, colsample_bytree=0.7, importance_type='split',\n",
       "        is_unbalance=True, learning_rate=0.1, max_depth=6,\n",
       "        min_child_samples=20, min_child_weight=....0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'min_child_samples': range(10, 50, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=5)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'num_leaves': 70,\n",
    "          'max_depth': 6,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False,  **params)\n",
    "\n",
    "min_child_samples_s = range(10,50,10) \n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "\n",
    "grid_search = GridSearchCV(lg, n_jobs=4,  param_grid=tuned_parameters, cv = kfold, scoring=\"roc_auc\", verbose=5, refit = False)\n",
    "grid_search.fit(x_train , y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8273816607159732\n",
      "{'min_child_samples': 40}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VNXdx/HPjwQS2UVAgQAJi7IJqKkrtghCERWqWAVXKhafp8W21tpKKy3FLmi1gK2lDwhCta2ilhqVqgTcQESCoKwSICwhCGERwQUM+T1/3Bs7xpgMITAzyff9euWVuWfOPXOOR+Y799w7uebuiIiIfJVase6AiIjENwWFiIiUS0EhIiLlUlCIiEi5FBQiIlIuBYWIiJRLQSEiIuVSUIiISLkUFCIiUq7kWHegKjRt2tTT09Nj3Q0RkYSydOnSXe7erKJ6UQWFmQ0AJgFJwMPuPr7U822AmUDjsM5d7j7HzPoB44E6wCHgTnefb2YNgNcjmkgDHnP3H5nZcOAPwLbwuT+7+8Pl9S89PZ2cnJxohiIiIiEz2xxNvQqDwsySgIeAfkA+sMTMstx9dUS1u4FZ7j7ZzLoAc4B0YBdwubsXmFk34EWglbvvB3pGvMZS4F8R7T3h7qOiGYCIiBxb0ZyjOBtY7+4b3f0Q8DgwuFQdBxqGjxsBBQDuvszdC8LyVUCqmaVE7mhmHYHmfPEIQ0RE4kQ0QdEK2BqxnR+WRRoLXG9m+QRHE7eV0c4QYJm7HyxVPozgCCLyz9gOMbN3zewpM2tdVqfMbKSZ5ZhZTmFhYRTDEBGRyogmKKyMstJ/m3wYMMPd04CBwKNm9nnbZtYVuBe4tYy2hgL/jNh+Fkh39+5ANsG5jy93wH2Ku2e6e2azZhWeixERkUqKJijygchP9WmES0sRRgCzANx9EZAKNAUwszRgNnCju2+I3MnMegDJ7r60pMzdd0ccdUwFzop6NCIiUuWiCYolQEczyzCzOgRHAFml6mwB+gKYWWeCoCg0s8bA88Bod19YRtvD+OLRBGbWImJzELAmmoGIiMixUeFVT+5eZGajCK5YSgKmu/sqMxsH5Lh7FnAHMNXMbidYlhru7h7u1wEYY2Zjwib7u/vO8PHVBEtVkX5gZoOAImAPMPzohigiIkfDqsOtUDMzM13foxCRmqTocDH3vfgew89Pp2XjEyrVhpktdffMiurpT3iIiCSYosPF/HjWO0x5bSPz1+6seIejpKAQEUkgRYeLuePJd8h6p4CfDejE9ee2PeavqaAQEUkQJSHxzPICfjrgNP63d/vj8roKChGRBHC42L8QEt/r3eG4vbaCQkQkzh0udu6YtZxnlhdw5zePb0iAgkJEJK4dLnZ+8uQ7/DsMie9fdHxDAhQUIiJxqyQkZi/bxk/6nxqTkAAFhYhIXDpc7NwZERKj+nSMWV8UFCIiceZwsXPnU+/wr2XbuKNfbEMCFBQiInHl85B4exs/7ncqt/WNbUiAgkJEJG4cLnZ++tS7n4fED+IgJEBBISISFw4XOz97+l2efjuf2y+On5AABYWISMwVFzt3Pf0uTy3N50cXd+SHF8dPSICCQkQkporDI4knw5D40cWnxrpLX6KgEBGJkciQ+GHf+AwJUFCIiMREcbFz17+CkPhB347c3i8+QwKiDAozG2Bm75nZejO7q4zn25jZy2a2zMzeNbOBYXk/M1tqZivC333C8gZmtjziZ5eZTQyfSzGzJ8LXWmxm6VU3XBGR2CsJiVk5YUjE2TmJ0iq8FaqZJQEPAf2AfGCJmWW5++qIancDs9x9spl1AeYA6cAu4HJ3LzCzbgS3U23l7vuBnhGvsRT4V7g5Atjr7h3MbChwL3DNUY5TRCQuFBc7o/+1IgiJPh24/eKOmFmsu1WuaI4ozgbWu/tGdz8EPA4MLlXHgYbh40ZAAYC7L3P3grB8FZBqZimRO5pZR6A58HpYNBiYGT5+Cuhr8f5fUUQkCsXFzs9nr+CJnK3c1qcDt/c7Ne5DAqILilbA1ojt/LAs0ljgejPLJziauK2MdoYAy9z9YKnyYcAT/t+bd3/+eu5eBOwDTirdmJmNNLMcM8spLCyMYhgiIrFTEhKPLwlC4scJEhIQXVCUNRIvtT0MmOHuacBA4FEz+7xtM+tKsIR0axltDQX+eYSvh7tPcfdMd89s1qxZBUMQEYmd4mLnF/8OQmLURYkVEhBdUOQDrSO20wiXliKMAGYBuPsiIBVoCmBmacBs4EZ33xC5k5n1AJLdfWlZr2dmyQRLWXuiHI+ISFwJQmIl/3xrK9+/qD139E+skIDogmIJ0NHMMsysDsERQFapOluAvgBm1pkgKArNrDHwPDDa3ReW0fYwvng0Qdj2TeHjq4D5EctSIiIJo7jYufuZlfzzrS18r3d7ftL/tIQLCYgiKMLzBKMIrlhaQ3B10yozG2dmg8JqdwDfNbN3CN74h4dv7qOADsCYiEthm0c0fzVfDoppwElmth74MfCly3FFROJdSUj8Y3EQEnd+MzFDAsCqw4f1zMxMz8nJiXU3RESAICTGPLOSvy/ewv/2bs9P4zQkzGypu2dWVE/fzBYRqULFxc4vs4KQ+J9vxG9IHAkFhYhIFXEPQuKxN4OQ+NmAxA8JUFCIiFQJ92C56bE3t3DrN9pVm5AABYWIyFFzd375zKogJL7ejrsGdKo2IQEKChGRo+Lu/CprFY++uTkIiUuqV0iAgkJEpNJKQuJvizYzspqGBCgoREQqxd0ZG4bEdy/MYHQ1DQlQUIiIHDF359fPrmbmos3c0iuDnw/sXG1DAhQUIiJHpCQkZryxiVt6ZfCLS6t3SICCQkQkapEhMaKGhAQoKEREouLujHsuCImbL8jg7hoSEqCgEBGpUElIPLIwCIkxl9WckAAFhYhIudyde55bwyMLN/GdC9JrXEiAgkJE5Cu5O795fg3TF+bxnQvS+eVlXWpcSICCQkSkTO7Ob59fw7QFeQw/v+aGBCgoRES+pCQkHg5D4leX19yQgCiDwswGmNl7ZrbezL50xzkza2NmL5vZMjN718wGhuX9zGypma0If/eJ2KeOmU0xs3VmttbMhoTlw82sMOKOeLdU1WBFRCri7vxujkIiUnJFFcwsCXgI6AfkA0vMLMvdV0dUu5vgFqmTzawLMAdIB3YBl7t7gZl1I7idaqtwn18AO939VDOrBTSJaO8Jdx91lGMTETki7s7v/7OWqa/ncdN5bRUSoQqDAjgbWO/uGwHM7HFgMBAZFA40DB83AgoA3H1ZRJ1VQKqZpbj7QeBmoFNYr5ggVEREYqIkJKa8tpEbz2vL2EFdFRKhaJaeWgFbI7bz+e9RQYmxwPVmlk9wNHFbGe0MAZa5+0EzaxyW3WNmb5vZk2Z2cmTdcAnrKTNrHdVIREQqyd0ZHxESv1ZIfEE0QVHWfy0vtT0MmOHuacBA4NFwOSlowKwrcC9wa1iUDKQBC939TGARcH/43LNAurt3B7KBmWV2ymykmeWYWU5hYWEUwxAR+TJ3Z/wLa/m/1zZyw7kKibJEExT5QOSn+jTCpaUII4BZAO6+CEgFmgKYWRowG7jR3TeE9XcDH4flAE8CZ4b77w6XpgCmAmeV1Sl3n+Lume6e2axZsyiGISLyRZ+HxKsbuf7cNowbrJAoSzRBsQToaGYZZlYHGApklaqzBegLYGadCYKiMFxieh4Y7e4LSyq7uxMcOfQOi/oSnvMwsxYR7Q4C1hzhmEREKuTu3PvCe5+HxD2DuykkvkKFJ7PdvcjMRhFcsZQETHf3VWY2Dshx9yzgDmCqmd1OsCw13N093K8DMMbMxoRN9nf3ncDPCJaoJgKFwHfC539gZoOAImAPMLyqBisiAkFI3Pfie/z11Q1cd04bxg1SSJTHgg/3iS0zM9NzcnJi3Q0RSQDuzh9efI+/vLKBa89pw28Gd6NWrZoZEma21N0zK6qnb2aLSI2hkKgcBYWI1Ajuzv0vBSEx7GyFxJFQUIhItefuPPDSOh56eQPDzm7Nb7+lkDgSCgoRqdZKQuLPL68PQ+J0hcQRUlCISLXl7vxxbhASQ7+mkKgsBYWIVEvuzoS56/jT/CAkfneFQqKyFBQiUu24OxOyc3lw/nquyVRIHC0FhYhUOxOyc3lwXi7XZLbm91cqJI6WgkJEqpUJc9fx4Lxcrs5MU0hUEQWFiFQbE+auY9K8XL59Vhrjr+yukKgiCgoRqRYmZv83JO4dopCoSgoKEUl4k7JzmZidy1UKiWNCQSEiCW1Sdi4Tstcx5EyFxLGioBCRhPXgvP+GxH1XdSdJIXFMKChEJCH9aV4uf5y7jivPbKWQOMYUFCKScP48P5cH5q7jyjNa8YereigkjrGogsLMBpjZe2a23szuKuP5Nmb2spktM7N3zWxgWN7PzJaa2Yrwd5+IfeqY2RQzW2dma81sSFieYmZPhK+12MzSq2aoIlId/Hl+Lve/FIbEtxUSx0OFt0I1syTgIaAfkA8sMbMsd18dUe1uYJa7TzazLsAcIB3YBVzu7gVm1o3gdqqtwn1+Aex091PNrBbQJCwfAex19w5mNhS4F7jmaAcqIonvoZfXc/9L67hCIXFcRXNEcTaw3t03uvsh4HFgcKk6DjQMHzcCCgDcfZm7F4Tlq4BUM0sJt28Gfh/WK3b3XWH5YGBm+PgpoK/pZrYiNd5DL6/nDy++xxVntOJ+hcRxFU1QtAK2Rmzn89+jghJjgevNLJ/gaOK2MtoZAixz94Nm1jgsu8fM3jazJ83s5NKv5+5FwD7gpGgGIyLVU0lIfKtnS4VEDEQTFGXNiJfaHgbMcPc0YCDwaLicFDRg1pVgCenWsCgZSAMWuvuZwCLg/iN4PcxspJnlmFlOYWFhFMMQkUT0l1eCkBjcsyUPXN1TIRED0QRFPtA6YjuNcGkpwghgFoC7LwJSgaYAZpYGzAZudPcNYf3dwMdhOcCTwJmlX8/MkgmWsvaU7pS7T3H3THfPbNasWRTDEJFEM/mVDdz3QhASf1RIxEw0QbEE6GhmGWZWBxgKZJWqswXoC2BmnQmCojBcYnoeGO3uC0squ7sDzwK9w6K+QMnJ8SzgpvDxVcD8sL6I1CB/fXUD976wlkE9WvKAlptiqsKrnty9yMxGEVyxlARMd/dVZjYOyHH3LOAOYKqZ3U6wTDTc3T3crwMwxszGhE32d/edwM8IlqgmAoXAd8Lnp4Xl6wmOJIZW2WhFJCH89dUNjP9PEBJ/vLoHyUn6ylcsWXX4sJ6Zmek5OTmx7oaIVIH/e3UDv//PWi7v0ZIJColjysyWuntmRfU0AyISN6a8ppCIR5oFEYkLU17bwO/mrOWy7i0UEnFGMyEiMTf1tY38bs5aLu3egonX9FRIxBnNhojE1MOvb+S3c9ZwafcWTFJIxCXNiIjEzMOvb+Q3z6/h0tMVEvFMsyIiMREZEhOHKiTimWZGRI67kpAYePopTBzak9oKibim2RGR42ragjx+8/waLul2CpOGnqGQSACaIRE5bqYvyOOe51ZzSbdTeHCYQiJRaJZE5LiYviCPcQqJhKSZEpFj7pGFQUgM6KqQSESaLRE5pmYszOPXz67mm11P5k/XKiQSkWZMRI6ZGQvzGFsSEsPOVEgkKM2aiBwTM9/YxNhnV9O/SxASdZL1dpOoNHMiUuX+tmgTv8paRb8uJ/PnaxUSiU6zJyJV6m+LNvHLZ4KQeEghUS1ENYNmNsDM3jOz9WZ2VxnPtzGzl81smZm9a2YDw/J+ZrbUzFaEv/tE7PNK2Oby8Kd5WD7czAojym+pqsGKyLH16CKFRHVU4a1QzSwJeAjoB+QDS8wsy91XR1S7G5jl7pPNrAswB0gHdgGXu3uBmXUjuJ1qq4j9rnP3sm5N94S7j6rUiEQkJh59czNjnlnFxZ0VEtVNNDN5NrDe3Te6+yHgcWBwqToONAwfNwIKANx9mbsXhOWrgFQzSzn6botIPHn0zc2M+fdKLu7cnL9cp5CobqKZzVbA1ojtfL54VAAwFrjezPIJjiZuK6OdIcAydz8YUfZIuLw0xswssm64hPWUmbWOoo8iEiOPRYTEQwqJaimaGbUyyrzU9jBghrunAQOBR83s87bNrCtwL3BrxD7XufvpwIXhzw1h+bNAurt3B7KBmWV2ymykmeWYWU5hYWEUwxCRqvb3xZu5+98r6dspCImU5KRYd0mOgWiCIh+I/FSfRri0FGEEMAvA3RcBqUBTADNLA2YDN7r7hpId3H1b+Hs/8A+CJS7cfXfEUcdU4KyyOuXuU9w9090zmzVrFsUwRKQq/X3xZn4xeyV9OjXnL9crJKqzaIJiCdDRzDLMrA4wFMgqVWcL0BfAzDoTBEWhmTUGngdGu/vCkspmlmxmJUFSG7gMWBlut4hodxCwpjIDE5Fj5x+Lt3weEpMVEtVehVc9uXuRmY0iuGIpCZju7qvMbByQ4+5ZwB3AVDO7nWBZari7e7hfB2CMmY0Jm+wPfAS8GIZEEsES09Tw+R+Y2SCgCNgDDK+isYpIFfjH4i38fPYKhUQNYu6lTzcknszMTM/JKesqWxGpSv98awuj/7WCi05rxl9vOEshkeDMbKm7Z1ZUT5cniEhUHo8IicnXKyRqkgqXnkSkZtu+7xOmL8hj6ut59A5DIrW2QqImUVCISJne2foB0xbkMWfFdordGXJmGr+9optCogZSUIjI5w4XO3NXv8+0BXks2bSXBinJDD8/nZvOT6d1k7qx7p7EiIJCRNj/6WfMyslnxht5bN3zCa2bnMAvL+vC1V9rTf0UvU3UdPo/QKQG27rnY2a8sYlZS7ay/2ARX0s/kV8M7Ey/LqeQVKusP8ogNZGCQqQGWrp5L9MWbOSFle9Ty4xLu7dgRK8Muqc1jnXXJA4pKERqiKLDxfxnZXD+YfnWD2iYmszIr7fnpvPb0qLRCbHunsQxBYVINbfvk894/K0tzHxjEwX7PiWjaT3uGdyVIWelUbeO3gKkYvq/RKSa2rz7Ix5ZuIlZOVv5+NBhzmt3EuMGd6NPp+bU0vkHOQIKCpFqxN15K28PDy/II3vNDpJrGZf3aMmIXhl0bdko1t2TBKWgEKkGDhUV8/yKAqYtyGPltg85sW5tRl3UgRvObUvzhqmx7p4kOAWFSAL74OND/H3xFv62aBM7PjxIh+b1+d0Vp3Plma30DWqpMgoKkQS0ofAA0xfk8fTb+Xz6WTEXdmzKvUO68/WOzXT+QaqcgkIkQbg7b2zYzbQFecxfu5M6ybW4omcrbu6VwWmnNIh196QaU1CIxLmDRYfJWh6cf1j7/n6a1q/D7RefynXntqFp/ZRYd09qgKiCwswGAJMI7kb3sLuPL/V8G2Am0Disc5e7zzGzfsB4oA5wCLjT3eeH+7wCtAA+CZvp7+47zSwF+BvBvbJ3A9e4+6ajGaRIItp94CCPvbmFR9/czK4DB+l0SgPuu6o7g3q01PkHOa4qDAozSwIeAvoB+cASM8ty99UR1e4GZrn7ZDPrAswB0oFdwOXuXmBm3Qhup9oqYr/r3L30relGAHvdvYOZDQXuBa6p3PBEEs+6HfuZ9noes5dv41BRMRed1owRvdpxQYeTMNP5Bzn+ojmiOBtY7+4bAczscWAwEBkUDjQMHzcCCgDcfVlEnVVAqpmluPvBcl5vMDA2fPwU8GczM68O92wV+QruzqvrCpm2II/Xc3eRWrsW3z4rje9ckEGH5vVj3T2p4aIJilbA1ojtfOCcUnXGAi+Z2W1APeDiMtoZAiwrFRKPmNlh4GngN2EYfP567l5kZvuAkwiOTkSqlU8/O8zsZduYtiCP9TsP0LxBCnd+8zSuPbsNJ9arE+vuiQDRBUVZx7qlP90PA2a4+wNmdh7wqJl1c/diADPrSrCE1D9in+vcfZuZNSAIihsIzk1E83qY2UhgJECbNm2iGIZI/Ni5/1MeXbSZvy/ewp6PDtG1ZUMmXNODS09vSZ1k3cpe4ks0QZEPtI7YTiNcWoowAhgA4O6LzCwVaArsNLM0YDZwo7tvKNnB3beFv/eb2T8Ilrj+FvF6+WaWTLCUtad0p9x9CjAFIDMzU8tSkhBWFexj2oI8nn2ngKJi5+LOJzOiVwbnZDTR+QeJW9EExRKgo5llANuAocC1pepsAfoCM8ysM5AKFJpZY+B5YLS7LyypHAZAY3ffZWa1gcuA7PDpLOAmYBFwFTBf5yckkRUXO/PX7mTagjwWbdxN3TpJXHdOW4afn05603qx7p5IhSoMivA8wSiCK5aSgOnuvsrMxgE57p4F3AFMNbPbCZaJhru7h/t1AMaY2Ziwyf7AR8CLYUgkEYTE1PD5aQRLV+sJjiSGVtVgRY6njw8V8fTSfB5ZuImNuz6iZaNURl/SiaFfa0OjurVj3T2RqFl1+LCemZnpOTmlr7IViY3t+z5h5hub+edbW9j3yWf0aN2YW3plMKDbKdRO0vkHiR9mttTdMyuqp29mi1SRd/M/YNqCPJ5/dzvF7gzodgojemVwZpsTdf5BEpqCQuQoHC525q7ewbQFG1myaS/1U5K56fx0hp+fTusmdWPdPZEqoaAQqYQDB4uYtWQrM97YxJY9H5N24gmMuawLV2em0SBV5x+kelFQiByB/L0fM/ONTTz+1lb2Hywis+2JjL6kE/27nkKS/ry3VFMKCpEoLN28l+kL8nhh1fsADDy9BSN6ZdCzdeMY90zk2FNQiHyFosPFvLDqfaYtyGPZlg9omJrMLRdmcNN56bRsfEKsuydy3CgoRErZ98lnPLFkCzPf2My2Dz4h/aS6jBvclSFnplEvRf9kpObR//Uioc27P+KRhZt4MmcrHx06zLntmjB2UFf6dGqu8w9SoykopEZzd5Zs2svDr29k7podJNcyLu/ekpt7ZdCtVaNYd08kLigopEY6VFTMnBXbmbYgjxXb9tG4bm2+37sDN5zXlpMbpsa6eyJxRUEhNcoHHx/iH29tYeYbm9jx4UHaN6vHb6/oxpVnpHFCHd1eVKQsCgqpETYWHmD6wjyeXrqNTz47zIUdmzJ+SHe+0bEZtXT+QaRcCgqpttydRRt28/CCPOav3UmdpFp864zg/EOnUxpW3ICIAAoKqYYOFh0ma3kB0xbksfb9/ZxUrw4/7NuR689tS7MGKbHunkjCUVBItbH7wEH+vngLf1u0mV0HDnLayQ24b0h3BvVsSWptnX8QqSwFhSS8dTv2M31BHrOXbeNgUTG9T2vGLb3acUGHk/TnvUWqQFRBYWYDgEkEd6N72N3Hl3q+DTATaBzWucvd55hZP2A8UAc4BNzp7vNL7ZsFtHP3buH2WOC7QGFY5efuPqdyw5Pqyt15LXcX0xbk8dq6QlKSazHkrDRuviCdDs0bxLp7ItVKhUFhZknAQ0A/IB9YYmZZ7r46otrdwCx3n2xmXYA5QDqwC7jc3QvMrBvB7VRbRbR9JXCgjJed4O73V3JMUo19+tlh/r1sG9MW5JG78wDNG6Twk/6ncu05bWlSr06suydSLUVzRHE2sN7dNwKY2ePAYCAyKBwouYykEVAA4O7LIuqsAlLNLMXdD5pZfeDHwEhg1lGNQqq9nfs/5bFFm3ls8Rb2fHSIri0b8sere3BZ95bUSdbtRUWOpWiCohWwNWI7HzinVJ2xwEtmdhtQD7i4jHaGAMvc/WC4fQ/wAPBxGXVHmdmNQA5wh7vvjaKfUg2tLviQaQvyePadAj4rLqZvp5MZ0SuDc9s10fkHkeMkmqAo61+jl9oeBsxw9wfM7DzgUTPr5u7FAGbWFbgX6B9u9wQ6uPvtZpZeqq3JBCHi/DdMbv5Sp8xGEhyN0KZNmyiGIYmiuNh5+b2dTFuQxxsbdnNC7SSGnt2a71yQQUbTerHunkiNE01Q5AOtI7bTCJeWIowABgC4+yIzSwWaAjvNLA2YDdzo7hvC+ucBZ5nZprAPzc3sFXfv7e47Sho1s6nAc2V1yt2nAFMAMjMzSweXJKDDxc6z7xTwp/m5bCj8iBaNUrnrkk4M+1obGtXV7UVFYiWaoFgCdDSzDGAbMBS4tlSdLUBfYIaZdQZSgUIzaww8D4x294Ulld19MsGRA+ERxXPu3jvcbuHu28OqVwArKzUySRiHi53n3i3gwXlBQHQ6pQGThvZk4OktqJ2k8w8isVZhULh7kZmNIrhiKQmY7u6rzGwckOPuWcAdwFQzu51gyWi4u3u4XwdgjJmNCZvs7+47y3nJ+8KlKQc2AbdWdnAS34qLnedWbOfBebms33mA005uwOTrzuSbXU/R318SiSPmnvirNpmZmZ6TkxPrbkiUioudOSu3Myk7l9ydBzj15Pr8sO+pXNJNASFyPJnZUnfPrKievpktx01xsfPCqveZlJ3Lezv206F5ff407AwuPb2FAkIkjiko5JgrLnZeXPU+k+blsvb9/bRvVo8Hw4DQLUZF4p+CQo4Zd+fFVTuYNC+XNds/pF2zekwa2pPLurdUQIgkEAWFVDl3Z+7qHUzMzmX19g/JaFqPCdf0YFCPVgoIkQSkoJAq4+5kr9nJxOx1rCr4kPST6vLHq3swqEdLknWZq0jCUlDIUXN35q/dycTsXFZs20fbk+py/7d78K2eCgiR6kBBIZXm7rzyXiETs9fxTv4+Wjc5gfuu6s6VZ7RSQIhUIwoKOWLuzivrCpmYncs7Wz8g7cQTuG9Id644s5W+SS1SDSkoJGolNwuaMHcdy7d+QKvGJzD+ytMZclaaAkKkGlNQSIXcnddzdzExex1vbwkC4vdXns6QM9N0LwiRGkBBIV/J3Vm4fjcTs9eRs3kvLRul8tsruvHts1orIERqEAWFfIm7s2jDbiZkr2PJpr20aJTKPd/qxtWZaaQkJ8W6eyJynCko5Ave2LCLidm5vJW3h1MapjJucFeu+VprBYRIDaagEADe3LibCXPXsThvDyc3TOHXg4KASK2tgBCp6RQUNdxbeXuYMHcdizbuplmDFH51eReGnd1GASEin1NQ1FBLNu1hYvY6Fq7fTdP6KYy5rAvXnaOAEJEvU1DUMEs372HC3FwWrN9F0/p1uPvSzlx3TltOqKOAEJGyRRUUZjYAmERwK9SH3X18qefbADOBxmGdu9x9jpn1A8bkCzI6AAAL5ElEQVQDdYBDwJ3uPr/UvllAO3fvFm43AZ4A0gluhXq1u++t7AAl8PaWvUyYu47Xc3dxUr06/GJgZ64/VwEhIhWrMCjMLAl4COgH5ANLzCzL3VdHVLsbmOXuk82sCzCH4I1+F3C5uxeYWTeC+263imj7SuBAqZe8C5jn7uPN7K5w+2eVHWBNt2zLXiZm5/LqukKa1KvD6Es6ccN5balbRweTIhKdaN4tzgbWu/tGADN7HBgMRAaFAw3Dx42AAgB3XxZRZxWQamYp7n7QzOoDPwZGArMi6g0GeoePZwKvoKA4Ysu3fsDE7HW88l4QEHdd0okbzm1LvRQFhIgcmWjeNVoBWyO284FzStUZC7xkZrcB9YCLy2hnCLDM3Q+G2/cADwAfl6p3srtvB3D37WbWvKxOmdlIgpChTZs2UQyjZng3/wMmZucyf+1OGtetzU8HnMZN56UrIESk0qJ59yjrlmReansYMMPdHzCz84BHzaybuxcDmFlX4F6gf7jdE+jg7rebWXplOu7uU4ApAJmZmaX7U+OsyN/HpHnryF4TBMSd3zyNm85Pp74CQkSOUjTvIvlA64jtNMKlpQgjgAEA7r7IzFKBpsBOM0sDZgM3uvuGsP55wFlmtinsQ3Mze8XdewM7zKxFeDTRAthZuaHVDCu37WNidi7Za3bQ6ITa/KT/qdx0fjoNUmvHumsiUk1EExRLgI5mlgFsA4YC15aqswXoC8wws85AKlBoZo2B54HR7r6wpLK7TwYmA4RHFM+FIQGQBdxEcLXUTcAzlRlYdbeqYB+TsnN5afUOGqYm8+N+pzL8gnQaKiBEpIpVGBTuXmRmowiuWEoCprv7KjMbB+S4exZwBzDVzG4nWJYa7u4e7tcBGGNmY8Im+7t7eUcJ44FZZjaCIIC+XenRVUNrtn/IxOx1vLhqBw1Sk/nRxR35zgUZNDpBASEix4a5J/7yfmZmpufk5MS6G8fU2vc/ZFJ2Lv9Z+T4NUpK5uVcGN/dSQIhI5ZnZUnfPrKieznTGuffe38+D83J5fsV26qck84M+HRjRqx2N6iogROT4UFDEqdwd+5k4L5c5K7ZTt3YSoy7qwC0XZtC4bp1Yd01EahgFRZxZv3M/k+at57l3C6hbO4nv9W7PLb3acWI9BYSIxIaCIk6s33mAP83PJeudAk6oncT/fKM9372wHU0UECISYwqKGNtYeIAH5wUBkZKcxMivt2Pkhe04qX5KrLsmIgIoKGImb9dH/GleLv9evo2U5CS+e2E7Rn5dASEi8UdBcZxt2vURf5q/nn8v30btJGNErwxu/UZ7miogRCROKSiOk827g4CYvWwbybWM75yfzq3faE+zBgoIEYlvCopjbMvuj/nzy7k8/XYQEDedl87/9G5H8wapse6aiEhUFBTHyNY9H/Pn+et5+u18atUybji3Ld/r3Z7mDRUQIpJYFBRVLH/vxzz08nqezAkC4vpz2/K/vdtzsgJCRBKUgqKKbPvgkzAgtmIY157Thu/17sApjRQQIpLYFBRHqSAMiFk5wU0Ar/laa77XuwMtG58Q456JiFQNBUUlbd/3CX95eQNPLNmK41yd2ZrvXdSBVgoIEalmFBRH6P19nzL5lfX8862tFLvz7czWfP+i9qSdWDfWXRMROSYUFFHa8eGnTH5lA/94awvFxc5VZ6Xx/Ys60LqJAkJEqreogsLMBgCTCO5w97C7jy/1fBtgJtA4rHOXu88xs34Ed6yrAxwC7nT3+eE+LwAtwj68Dnzf3Q+b2Vjgu0Bh2PzP3X3OUY3yKOz88FMmv7qBfyzeQlGxc9WZaYzqo4AQkZqjwqAwsyTgIaAfkA8sMbMsd18dUe1uYJa7TzazLsAcIB3YBVzu7gVm1o3gdqqtwn2udvcPzcyApwhuefp4+NwEd7//6IdXeTv3f8r/vbqRx97cTFGxc+UZrbitT0fanKSAEJGaJZojirOB9e6+EcDMHgcGA5FB4UDD8HEjoADA3ZdF1FkFpJpZirsfdPcPI/pQJ2wj5gr3H+T/Xt3AY4s3c6iomCvOSOO2Ph1Ib1ov1l0TEYmJaIKiFbA1YjsfOKdUnbHAS2Z2G1APuLiMdoYAy9z9YEmBmb1IEET/ITiqKDHKzG4EcoA73H1v6cbMbCQwEqBNmzZRDKN8uw4cZMprG/nbok0cKirmW+ERRIYCQkRquFpR1LEyykp/+h8GzHD3NGAg8KiZfd62mXUF7gVu/UIj7t8kOE+RAvQJiycD7YGewHbggbI65e5T3D3T3TObNWsWxTDKtvvAQX4/Zw0X3vsyD7++kUu6tSD7x9/gj1f3VEiIiBDdEUU+0DpiO41waSnCCGAAgLsvMrNUoCmw08zSgNnAje6+oXTj7v6pmWURLGfNdfcdJc+Z2VTguSMYzxF5YskWfv3saj757DCDe7Tktr4dad+s/rF6ORGRhBRNUCwBOppZBrANGApcW6rOFqAvMMPMOgOpQKGZNQaeB0a7+8KSymZWH2jg7tvNLJngKOT18LkW7r49rHoFsLLSo6tA6yZ1ubjzyfygb0c6NFdAiIiUpcKgcPciMxtFcMVSEjDd3VeZ2Tggx92zgDuAqWZ2O8Gy1HB393C/DsAYMxsTNtmfYDkry8xSwjbnA38Nn7/PzHqG7Wyi1HJVVTq/fVPOb9/0WDUvIlItmHtcXGx0VDIzMz0nJyfW3RARSShmttTdMyuqF83JbBERqcEUFCIiUi4FhYiIlEtBISIi5VJQiIhIuRQUIiJSLgWFiIiUq1p8j8LMCoHNldy9KcGfQ68ONJb4U13GARpLvDqasbR19wr/WF61CIqjYWY50XzhJBFoLPGnuowDNJZ4dTzGoqUnEREpl4JCRETKpaCAKbHuQBXSWOJPdRkHaCzx6piPpcafoxARkfLpiEJERMpVo4LCzKab2U4zWxlR1sTM5ppZbvj7xFj2MVpfMZaxZrbNzJaHPwNj2cdomFlrM3vZzNaY2Soz+2FYnnDzUs5YEnFeUs3sLTN7JxzLr8PyDDNbHM7LE2ZWJ9Z9LU8545hhZnkRc9Iz1n2NlpklmdkyM3su3D7mc1KjggKYQXjL1gh3AfPcvSMwL9xOBDP48lgAJrh7z/BnznHuU2UUAXe4e2fgXOD7ZtaFxJyXrxoLJN68HAT6uHsPgvvXDzCzc4F7CcbSEdhLcBvkePZV4wC4M2JOlseui0fsh8CaiO1jPic1Kijc/TVgT6niwcDM8PFM4FvHtVOV9BVjSTjuvt3d3w4f7yf4B9CKBJyXcsaScDxwINysHf440Ad4KiyP+3kpZxwJyczSgEuBh8Nt4zjMSY0Kiq9wcsk9usPfzWPcn6M1yszeDZem4n65JpKZpQNnAItJ8HkpNRZIwHkJlziWAzuBucAG4AN3Lwqr5JMAQVh6HO5eMie/DedkQnhb5kQwEfgpUBxun8RxmBMFRfUyGWhPcIi9HXggtt2JnpnVB54GfuTuH8a6P0ejjLEk5Ly4+2F37wmkAWcDncuqdnx7deRKj8PMugGjgU7A14AmwM9i2MWomNllwE53XxpZXEbVKp8TBQXsMLMWAOHvnTHuT6W5+47wH0UxMJXgH3fcM7PaBG+sf3f3f4XFCTkvZY0lUeelhLt/ALxCcN6lsZklh0+lAQWx6teRihjHgHCZ0N39IPAIiTEnFwCDzGwT8DjBktNEjsOcKCggC7gpfHwT8EwM+3JUSt5YQ1cAK7+qbrwI11inAWvc/Y8RTyXcvHzVWBJ0XpqZWePw8QnAxQTnXF4Grgqrxf28fMU41kZ8CDGCNf24nxN3H+3uae6eDgwF5rv7dRyHOalRX7gzs38CvQn+2uIO4FfAv4FZQBtgC/Btd4/7k8RfMZbeBMsbDmwCbi1Z549XZtYLeB1YwX/XXX9OsLafUPNSzliGkXjz0p3gxGgSwQfKWe4+zszaEXyabQIsA64PP5XHpXLGMR9oRrB0sxz4n4iT3nHPzHoDP3H3y47HnNSooBARkSOnpScRESmXgkJERMqloBARkXIpKEREpFwKChERKZeCQkREyqWgEBGRcikoRESkXP8P9ZdX/PfkaakAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.82344795, 0.82446397, 0.8253571 , 0.82738166])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_means"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第二次细调"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 2 candidates, totalling 10 fits\n",
      "[CV] min_child_samples=40 ............................................\n",
      "[CV] min_child_samples=40 ............................................\n",
      "[CV] min_child_samples=40 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] min_child_samples=40 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=40, score=0.8557378379923398, total=   4.6s\n",
      "[CV] ... min_child_samples=40, score=0.8117361230206368, total=   4.6s\n",
      "[CV] min_child_samples=40 ............................................\n",
      "[CV] min_child_samples=50 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=40, score=0.8064860581820926, total=   5.2s\n",
      "[CV] min_child_samples=50 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=40, score=0.8294663080375202, total=   5.9s\n",
      "[CV] min_child_samples=50 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=40, score=0.8334817162957552, total=   5.0s\n",
      "[CV] min_child_samples=50 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=50, score=0.8523345338135254, total=   5.4s\n",
      "[CV] min_child_samples=50 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "[Parallel(n_jobs=4)]: Done   6 out of  10 | elapsed:   12.5s remaining:    8.3s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... min_child_samples=50, score=0.8126511175898932, total=   5.4s\n",
      "[CV] ... min_child_samples=50, score=0.8122921476013579, total=   5.3s\n",
      "[CV] ... min_child_samples=50, score=0.8264468027565712, total=   2.5s\n",
      "[CV] .... min_child_samples=50, score=0.830415951344469, total=   2.1s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  10 out of  10 | elapsed:   15.3s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss',\n",
       "        categorical_feature=[0, 1, 3, 5, 6, 12, 15, 16, 17, 18, 19, 20],\n",
       "        class_weight=None, colsample_bytree=0.7, importance_type='split',\n",
       "        is_unbalance=True, learning_rate=0.1, max_depth=6,\n",
       "        min_child_samples=20, min_child_weight=....0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'min_child_samples': range(40, 60, 10)},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=5)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'num_leaves': 70,\n",
    "          'max_depth': 6,\n",
    "          'colsample_bytree': 0.7,\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False, **params)\n",
    "min_child_samples_s = range(40,60,10)\n",
    "tuned_parameters = dict( min_child_samples = min_child_samples_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='roc_auc',verbose=5, refit = False)\n",
    "grid_search.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8273816607159732\n",
      "{'min_child_samples': 40}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAD8CAYAAABpcuN4AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xd4lFX6xvHvkwQSepGASGhSpEoLKC1R6aigYgEVUBGwUCSuru7qLqvu/iy7AVQQsaFYKAqKAlIsCV0SQ28GpEQQgiBFFETO74+8urMxJgMkmSRzf64rV2bOnPfMcy50bt6Zdx7MOYeIiMgfCQl0ASIiUrApKEREJFsKChERyZaCQkREsqWgEBGRbCkoREQkWwoKERHJloJCRESypaAQEZFshfkzycy6A+OAUOBl59yTmR6vAbwOlPfmPOScm2tmXYAngeLASeAB59ynZlYGWOyzRBTwpnPuPp81rwdmAK2dc0nZ1VepUiVXq1Ytf7YiIiKe5OTkA865yJzm5RgUZhYKjAe6AGnAKjOb7Zzb6DPtEWC6c+4FM2sEzAVqAQeAq51ze8ysCTAfqOacOwo093mOZGCmz/0ywAhgZY47BWrVqkVSUrZZIiIimZjZTn/m+fPWUxsg1Tm33Tl3EpgK9M40xwFlvdvlgD0AzrkU59web3wDEGFm4ZkKrQdU5n/PMB4HngZ+8mcTIiKSd/wJimrAbp/7ad6Yr9HArWaWRsbZxPAs1ukDpDjnTmQa7wdMc153QjNrAVR3zn3kR20iIpLH/AkKy2Isc8vZfsBk51wU0BOYYma/rW1mjYGngKFZrNUXeMebFwKMAe7PsSizIWaWZGZJ6enpfmxDRETOhj9BkQZU97kfhffWko9BwHQA59xyIAKoBGBmUcAsYIBzbpvvQWbWDAhzziV7Q2WAJsDnZrYDuBSYbWbRmYtyzk1yzkU756IjI3P8LEZERM6SP0GxCqhnZrXNrDgZZwCzM83ZBXQCMLOGZARFupmVB+YADzvnlmaxdj+8swkA59xh51wl51wt51wtYAXQK6ernkREJO/kGBTOuVPAMDKuWNpExtVNG8zsMTPr5U27HxhsZmvIeOG/zfvMYRhQF3jUzFZ7P5V9lr8Rn6AQEZGCx4rCv3AXHR3tdHmsiMiZMbNk59zv3trPLKi/mb0t/Rj/WbCFn37+JdCliIgUWEEdFAs37uO5T1O58tnFJO88GOhyREQKpKAOirti6/D6HW346efTXD9xOaNnb+CHE6cCXZaISIES1EEBEFs/kvmjYhhwaU1eX76DrmMSSdyq72WIiPwq6IMCoHR4GP/o3YTpQ9sSXiyEAa9+wZ9mrOH74ycDXZqISMApKHy0rlWRuSM6cs9ldZiV8g2d4xOZt25voMsSEQkoBUUmEcVCebB7Az64tz2Vy4Rz91tfcvebyew/qv6EIhKcFBR/oEm1cnwwrD0Pdr+ITzbvp0t8IjOSdlMUvnciInImFBTZKBYawj2X1WXeyI7Ur1KaB95dy4BXv2D3weOBLk1EJN8oKPxQJ7I004a05bHejfly5yG6jU1k8tKvOX1aZxciUvQpKPwUEmIMaFuL+aNiiK5VkdEfbuSGF5eTuv9ooEsTEclTCoozFFWhJK/f3pr/3NCM1P3H6DluCeM/S+XnX04HujQRkTyhoDgLZkafVlEsioulc6PKPDN/C72fX8r6bw4HujQRkVynoDgHkWXCmXBLKybe2or0YyfoPX4pT328WU0GRaRIUVDkgu5NzmfRqFj6tKzGC59vo+e4xXzxtZoMikjRoKDIJeVKFuPp65vx5qBLOPnLaW58cTmPvr+eY2oyKCKFnIIil3WoV4n598Vwe/tavLlyJ13jE/hsy/5AlyUictYUFHmgVHgYf7+6Me/e1Y6S4WHc/toq4qat5tAPajIoIoWPgiIPtapZgTkjOjD8irrMXrOHLmMSmLN2r9qAiEihoqDIY+Fhodzf9SJmD+tA1XIluPftLxk6JZn9R9RkUEQKBwVFPml0QVlm3dOOh3s0IGFrOp3iE5i+Sk0GRaTgU1Dko7DQEIbG1mHeyI40rFqWB99by62vrGTXd2oyKCIFl4IiAC6MLM3UwZfyxDVNWLP7MN3GJvLKkq/5RU0GRaQAUlAESEiIceulNVkwKoZLLqzI4x9t5PqJy/hqn5oMikjBoqAIsAvKl+C121oz9qbm7DjwA1c+u4RnP/mKk6fUZFBECgYFRQFgZlzTohoL42Lp1uR84hdupdfzS1ib9n2gSxMRUVAUJJVKh/Ncvxa8NCCaQ8dPcs34pfzf3E38eFJNBkUkcPwKCjPrbmZbzCzVzB7K4vEaZvaZmaWY2Voz6+mNdzGzZDNb5/2+whsvY2arfX4OmNlY77G7vPmrzWyJmTXKzQ0XBl0aVWHBqFhual2dFxO302NcIiu2fxfoskQkSFlO1/GbWSiwFegCpAGrgH7OuY0+cyYBKc65F7wX9rnOuVpm1gLY55zbY2ZNgPnOuWpZPEcyMMo5l2hmZZ1zR7zxXsA9zrnu2dUYHR3tkpKSzmTfhcay1AM8NHMduw4e55ZLavBQjwaUiSgW6LJEpAgws2TnXHRO8/w5o2gDpDrntjvnTgJTgd6Z5jigrHe7HLAHwDmX4pzb441vACLMLDxTofWAysBi75gjPg+X8tYOWu3qVuLj+zpyZ4favPPFLrqOSeTTzfsCXZaIBBF/gqIasNvnfpo35ms0cKuZpQFzgeFZrNOHjLOOE5nG+wHTnM+pjZnda2bbgKeBEVkVZWZDzCzJzJLS09P92EbhVbJ4GI9c1Yj37m5HmYgw7picxH1TUzioJoMikg/8CQrLYizz3/L7AZOdc1FAT2CKmf22tpk1Bp4ChmaxVl/gnf9Z3Lnxzrk6wJ+BR7Iqyjk3yTkX7ZyLjoyM9GMbhV+LGhX4aHhHRnaqx5x1e+kcn8DsNXvUBkRE8pQ/QZEGVPe5H4X31pKPQcB0AOfcciACqARgZlHALGCAc26b70Fm1gwIc84l/8FzTwWu8aPGoFE8LIRRXerz4fAOVK9QghHvpDD4jWS+PawmgyKSN/wJilVAPTOrbWbFyTgDmJ1pzi6gE4CZNSQjKNLNrDwwB3jYObc0i7X7kelswvvM4ldXAl/5s5Fg0+D8ssy8pz1/7dmQJanpdIlP4J0vdunsQkRyXY5B4Zw7BQwD5gObgOnOuQ1m9ph3VRLA/cBgM1tDxgv/bd5nDsOAusCjPpfCVvZZ/kYyBQUwzMw2mNlqIA4YeC4bLMpCQ4zBMRfy8cgYGlcry8Mz13HzSyvZ+d0PgS5NRIqQHC+PLQyK8uWx/jp92jEtaTf/mrOJn0+f5k9dL+L29rUJDcnqIyYRkdy9PFYKgZAQo1+bGiyMi6VD3Uo8MWcT172wjC3fqsmgiJwbBUURc365CF4aEM2z/Vqw++BxrnpuMWMWblWTQRE5awqKIsjM6NXsAhbFxdKzaVXGffIVVz23mNW71WRQRM6cgqIIq1iqOOP6tuCVgdEc+fEU101YyhMfbVSTQRE5IwqKINCpYRUWxMXQt00NXl7yNd3GJrJs24FAlyUihYSCIkiUjSjGv65tyjuDLyXE4OaXVvLwzLUc+ennQJcmIgWcgiLItK1zHvNGxjA05kKmrdpNl/gEFm1Uk0ER+WMKiiBUongoD/dsyPv3tqdCyeLc+UYSw99J4cCxzP0aRUQUFEHt4qjyzB7Wgbgu9fl4/V66xCfwfso3agMiIv9DQRHkioeFMKJTPeaM6EjN80px37TVDHo9iT3f/xjo0kSkgFBQCAD1q5Thvbvb8ehVjVi+7Tu6jknkzRU7OX1aZxciwU5BIb8JDTEGdajN/PtiaFa9HI+8v55+L63g6wNqMigSzBQU8js1zivJm4Mu4ek+F7Nx7xG6j03kxYRtnPpFbUBEgpGCQrJkZtzYujqL4mKJqR/J/83bzLUTlrFxz5GcDxaRIkVBIdmqUjaCSf1bMf7mluw9/CO9nl/CfxZs4cQptQERCRYKCsmRmXHlxVVZOCqWXs0u4LlPU7ny2SUk7zwU6NJEJB8oKMRvFUoVJ/6m5rx2e2uOnzjF9ROX8Y8PN3D85KlAlyYieUhBIWfs8osqsyAulv6X1uS1pTvoOiaRJV+pyaBIUaWgkLNSOjyMx3o3YfrQthQLDeHWV1by4LtrOPyjmgyKFDUKCjknbWpXZN7Ijtx9WR3e+/IbusQnMH/Dt4EuS0RykYJCzllEsVD+3L0B79/TnvNKhzN0SjL3vvUl6UfVZFCkKFBQSK5pGlWO2cPa80C3i1i4cR+d4xN4LzlNTQZFCjkFheSqYqEh3Ht5XeaO7EDdyqW5f8YabnttFd+oyaBIoaWgkDxRt3IZZgxty+irG7Fqx0G6xifwxvIdajIoUggpKCTPhIQYt7XPaDLYsmYF/vbBBm6atJxt6ccCXZqInAEFheS56hVL8sYdbXjm+ovZ8u1ReoxbzITPU/lZTQZFCgW/gsLMupvZFjNLNbOHsni8hpl9ZmYpZrbWzHp6413MLNnM1nm/r/DGy5jZap+fA2Y21nsszsw2eut8YmY1c3PDEhhmxg3R1Vl0fyxXXFSZpz/ewjXjl7L+m8OBLk1EcpBjUJhZKDAe6AE0AvqZWaNM0x4BpjvnWgB9gQne+AHgaudcU2AgMAXAOXfUOdf81x9gJzDTOyYFiHbOXQy8Czx9LhuUgqVymQgm9m/FC7e0ZN+RE/Qev5Rn5m/mp5/VZFCkoPLnjKINkOqc2+6cOwlMBXpnmuOAst7tcsAeAOdcinNujze+AYgws3DfA82sHlAZWOwd85lz7rj38Aog6sy2JIVBj6ZVWRQXw7UtqjH+s230fHYxSTsOBrosEcmCP0FRDdjtcz/NG/M1GrjVzNKAucDwLNbpA6Q45zJ/C6sfMM1lfbH9IGCeHzVKIVS+ZHH+fUMz3rijDSd+Ps0NLy5n9OwN/HBCTQZFChJ/gsKyGMv8ot4PmOyciwJ6AlPM7Le1zawx8BQwNIu1+gLv/O5JzW4FooFnsizKbIiZJZlZUnp6uh/bkIIqpn4kC0bFMLBtLV5fntFkMGGr/kxFCgp/giINqO5zPwrvrSUfg4DpAM655UAEUAnAzKKAWcAA59w234PMrBkQ5pxLzjTeGfgr0CuLMxC855nknIt2zkVHRkb6sQ0pyEqFhzG6V2NmDG1LeLEQBr76BfdPX8P3x08GujSRoOdPUKwC6plZbTMrTsYZwOxMc3YBnQDMrCEZQZFuZuWBOcDDzrmlWazdj0xnE2bWAniRjJDYfyabkcIvulZF5o7oyL2X1+H91d/QOT6Reev2BroskaCWY1A4504Bw4D5wCYyrm7aYGaPmVkvb9r9wGAzW0PGC/9t3mcOw4C6wKM+l8JW9ln+Rn7/ttMzQGlghjc/cyhJERdRLJQHujVg9rD2VCkbzt1vfcldU5LZf+SnQJcmEpSsKDRsi46OdklJSYEuQ/LAqV9O89LirxmzaCsRYSE8elUjrm8VhVlWH52JyJkws2TnXHRO8/TNbCnQwkJDuPuyOswb2ZGLzi/DA++uZcCrX7D74PGcDxaRXKGgkEKhTmRppg1py+O9G/PlzkN0G5vI5KVf84uaDIrkOQWFFBohIUb/trWYPyqG1rUqMvrDjdz44nJS9x8NdGkiRZqCQgqdqAolmXx7a+JvbMa29GP0HLeE5z/9Sk0GRfKIgkIKJTPjupZRLBwVS5fGVfj3gq30el5NBkXygoJCCrXIMuGMv7klL/ZvxYFjGU0Gn5ynJoMiuUlBIUVCt8bns2hULNe3jGJiwjZ6jlvMF1+ryaBIblBQSJFRrmQxnrr+Yt4cdAknfznNjS8u59H313P0p58DXZpIoaagkCKnQ71KLBgVwx3ta/Pmyp10G5PIZ1vUDUbkbCkopEgqWTyMv13diHfvakep8DBuf20VcdNWc+gHNRkUOVMKCinSWtWswEcjOjDiirrMXrOHzvEJfLR2D0WhdY1IflFQSJEXHhZKXNeL+HB4By4oX4Jhb6cwdEoy+9RkUMQvCgoJGg2rlmXWPe14uEcDEram0zk+gWmrdunsQiQHCgoJKmGhIQyNrcPH98XQsGpZ/vzeOm55eSW7vlOTQZE/oqCQoFS7UimmDr6Uf17bhLVph+k2NpFXlqjJoEhWFBQStEJCjFsuqcnCuBja1jmPxz/aSJ8XlrF1n5oMivhSUEjQq1quBK8MjGZc3+bs/O4Hrnx2Mc9+8hUnT6nJoAgoKESAjCaDvZtXY1FcLN2bVCV+4VZ6Pb+ENbu/D3RpIgGnoBDxcV7pcJ7r14KXBkRz6PhJrp2wlH/N3cSPJ9VkUIKXgkIkC10aVWFhXCw3ta7OpMTt9BiXyPJt3wW6LJGAUFCI/IGyEcX4v+su5u07L+G0g34vreAvs9ZxRE0GJcgoKERy0K5uJebfF8PgjrWZ+sUuusYn8unmfYEuSyTfKChE/FCieCh/vbIRM+9pT7kSxbhjchIjp6bw3bETgS5NJM8pKETOQPPq5flweAfu61yPuev20mVMIrPXqMmgFG0KCpEzVDwshPs61+ej4R2pXrEkI95JYfAbSew9/GOgSxPJEwoKkbN00fllmHl3Ox65siFLUg/QNT6Rt1fu4rTagEgRo6AQOQehIcadHS9k/n0xNKlWjr/MWsfNL69gx4EfAl2aSK7xKyjMrLuZbTGzVDN7KIvHa5jZZ2aWYmZrzaynN97FzJLNbJ33+wpvvIyZrfb5OWBmY73HYszsSzM7ZWbX5+ZmRfJKzfNK8fbgS3jyuqZs+OYI3ccl8lLidjUZlCIhx6Aws1BgPNADaAT0M7NGmaY9Akx3zrUA+gITvPEDwNXOuabAQGAKgHPuqHOu+a8/wE5gpnfMLuA24O1z2ZhIfjMz+rapwcK4WDrUrcQ/527iuglL2fKtmgxK4ebPGUUbINU5t905dxKYCvTONMcBZb3b5YA9AM65FOfcHm98AxBhZuG+B5pZPaAysNg7Zodzbi2gjmxSKJ1fLoKXBkTzXL8WpB36kaueW8yYhVs5cUptQKRw8icoqgG7fe6neWO+RgO3mlkaMBcYnsU6fYAU51zmC8/7AdOcri+UIsTMuLrZBSyMi+XKplUZ98lXXP3cElJ2HQp0aSJnzJ+gsCzGMr+o9wMmO+eigJ7AFDP7bW0zaww8BQzNYq2+wDv+letTlNkQM0sys6T09PQzPVwkX1QsVZyxfVvw6m3RHP3pFNe9sIzHP9rI8ZOnAl2aiN/8CYo0oLrP/Si8t5Z8DAKmAzjnlgMRQCUAM4sCZgEDnHPbfA8ys2ZAmHMu+UwLd85Ncs5FO+eiIyMjz/RwkXx1RYMqLBgVwy2X1OCVJV/TfexilqUeCHRZIn7xJyhWAfXMrLaZFSfjDGB2pjm7gE4AZtaQjKBIN7PywBzgYefc0izW7sdZnE2IFEZlIorxxDVNmTrkUkIMbn55JQ+9t5bDP6rJoBRsOQaFc+4UMAyYD2wi4+qmDWb2mJn18qbdDww2szVkvPDf5n3mMAyoCzzqcylsZZ/lbyRTUJhZa++zjhuAF81swznuUaRAufTC8/j4vhiGxl7I9KTddB2TwMKNajIoBZcVhc+Qo6OjXVJSUqDLEDlja9O+58F317L526NcdXFVRvdqTKXS4TkfKJILzCzZORed0zx9M1skgC6OKs/sYR24v0t9FmzYR+f4BGalpKnJoBQoCgqRACseFsLwTvWYM6IDtSuVYtS0NdwxeRV7vleTQSkYFBQiBUS9KmV49652/O2qRqzYfpCuYxKZsmKnmgxKwCkoRAqQ0BDjjg61WTAqhubVy/Po++vp+9IKvlaTQQkgBYVIAVS9YkmmDGrD030uZtPeI3Qfm8jEhG2c+kWdbST/KShECigz48bW1VkUF0ts/UienLeZaycsY+OeI4EuTYKMgkKkgKtSNoIX+7diwi0t2Xv4R3o9v4T/LNiiJoOSbxQUIoWAmdGzaVUWjoqlV/MLeO7TVK58dgnJO9VkUPKegkKkEKlQqjjxNzZn8u2t+fHkL1w/cRn/+HADP5xQk0HJOwoKkULososqM39UDP0vrclrS3fQbWwii79SF2XJGwoKkUKqdHgYj/VuwvShbSkeGkL/V77gwXfXcPi4mgxK7lJQiBRybWpXZO7Ijtx9WR3e+/IbOo9J4OP13wa6LClCFBQiRUBEsVD+3L0BH9zbnsjS4dz1ZjL3vJXM/qM/Bbo0KQIUFCJFSJNq5fhgWHse6HYRizbtp0t8Iu8lq8mgnBsFhUgRUyw0hHsvr8vcER2pW7k0989Yw8DXVpF26HigS5NCSkEhUkTVrVyaGUPb8o9ejUnacZBuYxJ5Y/kONRmUM6agECnCQkKMge1qMf++GFrWrMDfPtjATZOWsy39WKBLk0JEQSESBKpXLMkbd7Th3zc0Y+u+Y/QYt5jxn6Xys5oMih8UFCJBwsy4vlUUC+Ni6NywMs/M38I145ey/pvDgS5NCjgFhUiQqVwmggm3tGLirS3Zd+QEvccv5emPN/PTz2oyKFlTUIgEqe5NqvJJXCzXtajGhM+30fPZxSTtOBjosqQAUlCIBLFyJYvxzA3NeOOONpz4+TQ3vLicv3+wnmNqMig+FBQiQkz9SBaMimFg21q8sWIn3cYkkrBVTQYlg4JCRAAoFR7G6F6NefeutkQUC2Hgq18QN3013x8/GejSJMAUFCLyP1rVrMicER0ZdnldZq/eQ+f4BOau2xvosiSAFBQi8jsRxUL5U7eL+GBYe84vF8E9b33JXVOS2X9ETQaDkYJCRP5Q4wvK8f497flz9wZ8umU/neMTmJ60W00Gg4xfQWFm3c1si5mlmtlDWTxew8w+M7MUM1trZj298S5mlmxm67zfV3jjZcxstc/PATMb6z0WbmbTvOdaaWa1cm+7InKmwkJDuPuyOnw8siMNzi/Lg++uZcCrX7D7oJoMBoscg8LMQoHxQA+gEdDPzBplmvYIMN051wLoC0zwxg8AVzvnmgIDgSkAzrmjzrnmv/4AO4GZ3jGDgEPOubrAGOCpc9mgiOSOCyNLM3XIpTzeuzFf7jxEt7GJvLb0a35Rk8Eiz58zijZAqnNuu3PuJDAV6J1pjgPKerfLAXsAnHMpzrk93vgGIMLMwn0PNLN6QGVgsTfUG3jdu/0u0MnMzP8tiUheCQkx+retxYK4WNrUrsg/PtzIDROXkbr/aKBLkzzkT1BUA3b73E/zxnyNBm41szRgLjA8i3X6ACnOuROZxvsB09x/3/T87fmcc6eAw8B5ftQpIvmkWvkSvHZba8bc1IztB36g57glPP/pV2oyWET5ExRZ/W0+87lmP2Cycy4K6AlMMbPf1jazxmS8hTQ0i7X6Au+c4fNhZkPMLMnMktLT9cUgkfxmZlzbIopFcbF0aVyFfy/YytXPLWFdmpoMFjX+BEUaUN3nfhTeW0s+BgHTAZxzy4EIoBKAmUUBs4ABzrltvgeZWTMgzDmXnNXzmVkYGW9l/a4BjXNuknMu2jkXHRkZ6cc2RCQvVCodzvibW/Ji/1Yc/OEk10xYypPz1GSwKPEnKFYB9cystpkVJ+MMYHamObuATgBm1pCMoEg3s/LAHOBh59zSLNbux/+eTeCtPdC7fT3wqdO1eCIFXrfG57MwLpbrW0YxMWEbPcYtZuX27wJdluSCHIPC+5xgGDAf2ETG1U0bzOwxM+vlTbsfGGxma8h44b/Ne3EfBtQFHvW5FLayz/I38vugeAU4z8xSgTjgd5fjikjBVK5EMZ66/mLeuvMSTp0+zU2TVvDI++s4+tPPgS5NzoEVhb+sR0dHu6SkpECXISI+jp88xX8WbOXVpV9TtWwE/7y2KZc3qJzzgZJvzCzZORed0zx9M1tE8kTJ4mE8elUj3ru7HaXCw7h98ipGTVvNwR/UZLCwUVCISJ5qWaMCH43owIhO9fhwzR66xCfw0do9agNSiCgoRCTPhYeFEtelPh8O70C1CiUY9nYKQ6Yks09NBgsFBYWI5JuGVcsy8+52/KVnAxK3ptM5PoGpX+zS2UUBp6AQkXwVFhrCkJg6zL8vhkZVy/LQzHXc8vJKdn2nJoMFlYJCRAKiVqVSvDP4Uv51bVPWph2m69gEXl68XU0GCyAFhYgETEiIcfMlNVgYF0O7OpV4Ys4m+rywjK371GSwIFFQiEjAVS1XglcGRjOub3N2HTzOlc8uZtyirzh5Sk0GCwIFhYgUCGZG7+bVWDgqhh5NqjJm0VZ6Pb+ENbu/D3RpQU9BISIFynmlw3m2XwteHhDN98d/5toJS/nX3E38eFJNBgNFQSEiBVLnRlVYEBdD3zY1mJS4ne7jElm+TU0GA0FBISIFVtmIYvzr2qa8PfgSAPq9tIKHZ67jiJoM5isFhYgUeO3qVOLjkTEMibmQaat20TU+kU827Qt0WUFDQSEihUKJ4qH8pWdDZt7TnnIlijHo9SRGvJPCd8cy/+vKktsUFCJSqDSvXp4Ph3dgVOf6zFu/ly5jEvlg9TdqA5KHFBQiUugUDwthZOd6zBnRkRoVSzJy6mrufD2JvYd/DHRpRZKCQkQKrfpVyvDe3e145MqGLN12gC7xiby1cien1QYkVykoRKRQCw0x7ux4IQvui+XiqHL8ddZ6bn55BTsO/BDo0ooMBYWIFAk1zivJW3dewpPXNWXDN0foNjaRSYnbOPWL2oCcKwWFiBQZZkbfNjVYGBdLx3qR/GvuZvq8sIzN3x4JdGmFmoJCRIqc88tF8NKAVjx/cwvSDv3IVc8uIX7hVk6cUhuQs6GgEJEiycy46uILWBQXy9XNLuDZT77i6ueWkLLrUKBLK3QUFCJSpFUoVZwxNzXntdtac/SnU1z3wjIe/2gjx0+eCnRphYaCQkSCwuUNKrNgVAy3XFKDV5Z8TbexiSxNPRDosgoFBYWIBI0yEcV44pqmTBtyKWEhIdzy8koeem8th39Uk8HsKChEJOhccuF5zBvZkaGxFzI9aTdd4hNYsOHbQJdVYCkoRCQoRRQL5eEeDXn/3vZULFWcIVOSGfb2lxxQk8Hf8SsozKy7mW0xs1QzeyiLx2uY2WdmlmJma82spzdQkaKcAAALfUlEQVTexcySzWyd9/sKn2OKm9kkM9tqZpvNrI83XtPMPvHW+dzMonJrsyIimV0cldFk8E9d67Ngwz46xycwKyVNTQZ95BgUZhYKjAd6AI2AfmbWKNO0R4DpzrkWQF9ggjd+ALjaOdcUGAhM8Tnmr8B+51x9b90Eb/zfwBvOuYuBx4D/O5uNiYj4q1hoCMOuqMfckR24sFIpRk1bw+2TV/HN92oyCP6dUbQBUp1z251zJ4GpQO9McxxQ1rtdDtgD4JxLcc7t8cY3ABFmFu7dvwMvBJxzp51zv15+0Aj4xLv9WRbPJSKSJ+pWLsOMu9rx96sbsXL7QbrGJzBlhZoM+hMU1YDdPvfTvDFfo4FbzSwNmAsMz2KdPkCKc+6EmZX3xh43sy/NbIaZVfHG1nhzAa4FypjZeZkXM7MhZpZkZknp6el+bENEJGehIcbt7WuzYFQMLWpU4NH319N30gq2px8LdGkB409QWBZjmeO1HzDZORcF9ASmmNlva5tZY+ApYKg3FAZEAUudcy2B5WS85QTwJyDWzFKAWOAb4HffjHHOTXLORTvnoiMjI/3YhoiI/6pXLMmUQW14+vqL2fztEXqMW8zEhOBsMuhPUKQB1X3uR+G9teRjEDAdwDm3HIgAKgF4H0bPAgY457Z5878DjnvjADOAlt7xe5xz13mfd/zVGzt8ZtsSETl3ZsaN0dVZFBfLZRdF8uS8zVwzYSkb9wRXk0F/gmIVUM/MaptZcTI+rJ6dac4uoBOAmTUkIyjSvbeY5gAPO+eW/jrZZVxO8CFwmTfUCdjoHV/J52zkYeDVs9iXiEiuqVw2ghf7R/PCLS359vAJej2/hH/P38JPPwdHk8Ecg8I5dwoYBswHNpFxddMGM3vMzHp50+4HBpvZGuAd4DYvDIYBdYFHzWy191PZO+bPwGgzWwv099aAjPDYYmZbgSrAP3NjoyIi56pH06osiouhd/NqPP9ZKlc+u5jknQcDXVaes6JwrXB0dLRLSkoKdBkiEkQStqbzl5nr2HP4Rwa2rcUD3S6iVHhYoMs6I2aW7JyLzmmevpktInIWYutHMn9UDAMurcnkZTvoNjaRxV8VzSswFRQiImepdHgY/+jdhBl3taV4WAj9X/mCB2as4fDxotVkUEEhInKOWteqyNwRHbnnsjrMTPmGzmMS+Hj93kCXlWsUFCIiuSCiWCgPdm/AB/e2J7J0OHe9+SV3v5nM/qM/Bbq0c6agEBHJRU2qleODYe15oNtFfLJ5P13iE3k3uXA3GVRQiIjksmKhIdx7eV3mjuhIvcql+dOMNQx8bRVph44HurSzoqAQEckjdSuXZvrQtvyjV2OSdhyk65hEXl+2o9A1GVRQiIjkoZAQY2C7WiwYFUN0rYr8ffYGbnxxOan7C0+TQQWFiEg+iKpQktdvb81/bmjGV/uP0XPcYsZ/lsrPhaDJoIJCRCSfmBl9WkWxKC6Wzo0q88z8LfR+finrvynYfU8VFCIi+SyyTDgTbmnFxFtbkn7sBL3HL+WpjzcX2CaDCgoRkQDp3qQqi0bFcl2Larzw+TZ6jlvMqh0Fr8mggkJEJIDKlSzGMzc0Y8qgNpz85TQ3TFzO3z5Yz7ETv/v32gJGQSEiUgB0rBfJ/PtiuL19Laas2Em3MYl8vmV/oMsCFBQiIgVGqfAw/n51Y969qx0liody22uriJu+mkM/nAxoXQoKEZECplXNCswZ0YHhV9Rl9uo9dBmTwNx1ewPWBkRBISJSAIWHhXJ/14uYPawDVcuV4J63vuSuN5PZfyT/mwwqKERECrBGF5Rl1j3teKhHAz7fkk7n+ASmJ+3O17MLBYWISAEXFhrCXbF1mDeyIw2qluXBd9fS/5Uv2H0wf5oMKihERAqJCyNLM3XwpTxxTRNW7/6ermMS+XDNnjx/XgWFiEghEhJi3HppTRaMiqF93UrUrlQqz58zLM+fQUREct0F5Uvw8sDofHkunVGIiEi2FBQiIpItBYWIiGRLQSEiItnyKyjMrLuZbTGzVDN7KIvHa5jZZ2aWYmZrzaynN97FzJLNbJ33+wqfY4qb2SQz22pmm82sT3ZriYhIYOR41ZOZhQLjgS5AGrDKzGY75zb6THsEmO6ce8HMGgFzgVrAAeBq59weM2sCzAeqecf8FdjvnKtvZiFAxRzWEhGRAPDn8tg2QKpzbjuAmU0FegO+QeGAst7tcsAeAOdcis+cDUCEmYU7504AdwANvHmnyQiVP1xLREQCw5+3nqoBu33up/Hfs4JfjQZuNbM0Ms4AhmexTh8gxTl3wszKe2OPm9mXZjbDzKqcwVoiIpJP/DmjsCzGMnej6gdMds79x8zaAlPMrIl3poCZNQaeArr6PG8UsNQ5F2dmccC/gf45rfVbUWZDgCHe3WNmtsWPvWSlEv89mwkW2nNw0J6Dw7nsuaY/k/wJijSgus/9KH7/dtAgoDuAc265mUWQUfx+M4sCZgEDnHPbvPnfAce9cYAZ3hrZruX7hM65ScAkP+rPlpklOefy5+uNBYT2HBy05+CQH3v2562nVUA9M6ttZsWBvsDsTHN2AZ0AzKwhEAGke28xzQEeds4t/XWyy+iP+yFwmTfUif9+5pHlWme8MxERyRU5BoVz7hQwjIwrljaRcUXSBjN7zMx6edPuBwab2RrgHeA2LwyGAXWBR81stfdT2Tvmz8BoM1tLxltO9+ewloiIBIAF+2uwmQ3x3sYKGtpzcNCeg0N+7Dnog0JERLKnFh4iIpKtoAsKMwv12oN85N2vbWYrzewrM5vmfWBfpGSx57e8lizrzexVMysW6BpzW+Y9+4w/Z2bHAlVXXsriz9nM7J9em5xNZjYi0DXmtiz23Mn7btZqM1tiZnUDXWNuMrMdXkuk1WaW5I1VNLOF3mvYQjOrkNvPG3RBAYwk40P5Xz0FjHHO1QMO8d/LdIuSzHt+i4xvxTcFSgB3BqKoPJZ5z5hZNFA+6+lFQuY930bGpe0NnHMNgamBKCqPZd7zC8AtzrnmwNtktAQqai53zjX3uST2IeAT7zXsE+9+rgqqoPC+03El8LJ334ArgHe9Ka8D1wSmuryRec8Azrm5zgN8QcZ3Y4qMrPbs9Sx7BngwUHXlpaz2DNwNPPbrl1Wdc/uzOraw+oM9B2MLoN5kvHZBHr2GBVVQAGPJeKH49Vve5wHfe5cAQ9btSQq7zHv+jfeWU3/g4/wuKo9ltedhwGzn3N7AlJTnstpzHeAmM0sys3lmVi8wpeWZrPZ8JzDXawHUH3gyEIXlIQcs8Lpx/9qZosqv/117vyv/4dFnKWiCwsyuIqNbbbLvcBZTi8xlYH+wZ18TgETn3OJ8LCtPZbVnM7sAuAF4LmCF5aFs/pzDgZ+8tyheAl7N9+LySDZ7HgX0dM5FAa8B8fleXN5q75xrCfQA7jWzmPx4Un9aeBQV7YFe3r9vEUHG6elYoLyZhXlnFVm1JynMfrdnM3vTOXermf0diASGBrTC3JfVn/MG4ASQmvFuIyXNLNU5V1Q+6Mzyz5mMM+T3vDmzyHjhLCqy2vMcMj6PWenNmUYRO1t2zv3amXu/mc0io7v3PjOr6pzba2ZVydTuKLeeOOh+yGgd8pF3ewbQ17s9Ebgn0PXlw57vBJYBJQJdV37tOdP4sUDXlk9/zk8Cd/iMrwp0fXm5ZzL+4nsAqO+NDwLeC3R9ubjPUkAZn9vLyOiL9wzwkDf+EPB0bj93MJ1R/JE/A1PN7AkgBXglwPXkh4nATmC59zfsmc65xwJbkuSBJ4G3zGwUcIyieXXbb5xzp8xsMPCemZ0m4yrGOwJcVm6qAszy/p8NA952zn1sZquA6WY2iIxeeTfk9hPrm9kiIpKtoPkwW0REzo6CQkREsqWgEBGRbCkoREQkWwoKERHJloJCRESypaAQEZFsKShERCRb/w+j5KvZBnll8QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = min_child_samples_s\n",
    "\n",
    "plt.plot(x_axis, test_means)\n",
    "#plt.errorbar(x_axis, -test_scores, yerr=test_stds ,label = ' Test')\n",
    "#plt.errorbar(x_axis, -train_scores, yerr=train_stds,label =  +' Train')\n",
    "\n",
    "plt.show()\n",
    "#  min_child_samples=40"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 列采样参数 sub_feature/feature_fraction/colsample_bytree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 9 candidates, totalling 45 fits\n",
      "[CV] colsample_bytree=0.1 ............................................\n",
      "[CV] colsample_bytree=0.1 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] colsample_bytree=0.1 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] colsample_bytree=0.1 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.1, score=0.8193647744812211, total=   3.1s\n",
      "[CV] colsample_bytree=0.1 ............................................\n",
      "[CV] ... colsample_bytree=0.1, score=0.7753092093980449, total=   3.1s\n",
      "[CV] colsample_bytree=0.2 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.1, score=0.8252028423410744, total=   3.4s\n",
      "[CV] colsample_bytree=0.2 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.1, score=0.7849113708110803, total=   4.8s\n",
      "[CV] colsample_bytree=0.2 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.1, score=0.8051041060680291, total=   3.3s\n",
      "[CV] colsample_bytree=0.2 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.2, score=0.8228040930657978, total=   4.3s\n",
      "[CV] colsample_bytree=0.2 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.2, score=0.8070482586876957, total=   3.8s\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.2, score=0.7980110901503459, total=   6.1s\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.2, score=0.8354568779520671, total=   4.5s\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8309890813468244, total=   4.0s\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "[Parallel(n_jobs=4)]: Done  10 tasks      | elapsed:   15.8s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... colsample_bytree=0.2, score=0.81870112028754, total=   5.4s\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8036485451323386, total=   4.5s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8075559199130116, total=   5.4s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8354614694826217, total=   3.9s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8181191437897483, total=   4.9s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .... colsample_bytree=0.4, score=0.848469101926485, total=   4.5s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.4, score=0.8216171039844509, total=   4.7s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.4, score=0.7991805387383246, total=   5.5s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.4, score=0.8252768807712669, total=   5.2s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.4, score=0.8368928791330088, total=   5.7s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8504405190647688, total=   4.4s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8145629680443606, total=   5.5s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8049904013153456, total=   5.5s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .... colsample_bytree=0.5, score=0.834410238562153, total=   5.3s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8241950013843464, total=   5.5s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.6, score=0.8089855370719716, total=   5.0s\n",
      "[CV] colsample_bytree=0.6 ............................................\n",
      "[CV] ... colsample_bytree=0.6, score=0.8475788029497513, total=   6.2s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.6, score=0.8114303812375729, total=   5.7s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... colsample_bytree=0.6, score=0.82667787153173, total=   5.8s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .... colsample_bytree=0.6, score=0.826158684214272, total=   4.7s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.7, score=0.8557378379923398, total=   4.9s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.7, score=0.8117361230206368, total=   5.6s\n",
      "[CV] colsample_bytree=0.8 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.7, score=0.8064860581820926, total=   6.1s\n",
      "[CV] colsample_bytree=0.8 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.7, score=0.8334817162957552, total=   4.8s\n",
      "[CV] colsample_bytree=0.8 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.7, score=0.8294663080375202, total=   5.6s\n",
      "[CV] colsample_bytree=0.8 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.8, score=0.8421398273595153, total=   5.0s\n",
      "[CV] colsample_bytree=0.8 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.8, score=0.8178352483850684, total=   6.7s\n",
      "[CV] colsample_bytree=0.9 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.8, score=0.7952369028549082, total=   6.0s\n",
      "[CV] colsample_bytree=0.9 ............................................\n",
      "[CV] ... colsample_bytree=0.8, score=0.8283665216814369, total=   6.0s\n",
      "[CV] colsample_bytree=0.9 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.8, score=0.8298953865678447, total=   5.3s\n",
      "[CV] colsample_bytree=0.9 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.9, score=0.8416451151889327, total=   6.1s\n",
      "[CV] colsample_bytree=0.9 ............................................\n",
      "[CV] ... colsample_bytree=0.9, score=0.8214917967186874, total=   4.9s\n",
      "[CV] ... colsample_bytree=0.9, score=0.7963653057000981, total=   4.8s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.9, score=0.8247078753472918, total=   5.7s\n",
      "[CV] ... colsample_bytree=0.9, score=0.8303243503099054, total=   1.2s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  45 out of  45 | elapsed:  1.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss',\n",
       "        categorical_feature=[0, 1, 3, 5, 6, 12, 15, 16, 17, 18, 19, 20],\n",
       "        class_weight=None, colsample_bytree=1.0, importance_type='split',\n",
       "        is_unbalance=True, learning_rate=0.1, max_depth=6,\n",
       "        min_child_samples=40, min_child_weight=....0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=5)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'num_leaves': 70,\n",
    "          'max_depth': 6,\n",
    "          #'colsample_bytree': 0.7,\n",
    "          'min_child_samples':40\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False, **params)\n",
    "#colsample_bytree_s = [i/10 for i in range(5,10)]\n",
    "#第一次使用5-10，第二次继续细调3-6\n",
    "colsample_bytree_s = [i/10 for i in range(1,10)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='roc_auc',verbose=5, refit = False)\n",
    "grid_search.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8273816607159732\n",
      "{'colsample_bytree': 0.7}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/sklearn/utils/deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8FfW9//HXJ3vYkbDJIosgEBapAVxa3EAQ11Zr3aVV6ab3Vm1dbm3r1eu1621t68+Ku3ax1NpKA4JFcS0qUSCQALJLSAgBJCxJSHLy+f1xBnuMQE4gyTnJeT8fj/NgZvKdOZ9zgHlnvjPfGXN3REREkmJdgIiIxAcFgoiIAAoEEREJKBBERARQIIiISECBICIigAJBREQCCgQREQEUCCIiEkiJdQGNkZWV5QMGDIh1GSIircr777+/3d27N9SuVQXCgAEDyMvLi3UZIiKtipltiqaduoxERARQIIiISECBICIigAJBREQCCgQREQEUCCIiElAgiIgIoEAQkRa2bPMuXly6BT2+N/60qoFpItJ6VVaH+L9/rubxtzZQ51BQvJu7zh2GmcW6NAkoEESk2b23YSe3P7+MjTsquHJCfwyY+cZ6KqtD/PeF2SQlKRTigQJBRJrNvv21/HTeKp5etIl+x2TyxxsmcOrxWbg77dNTwqFQE+Inl4wmWaEQcwoEEWkWb6/dzh1/zWfLrkqmnzqA7005gfbp4V2OmXHXucPITE3mwVfWUFUT4pdfOZHUZJ3WjCUFgog0qd1VNTwwdyV/em8zg7LaM+vrpzBuwDGfaWdm3DJ5KO3SknngpVXsr63jt1eOJT0lOQZVCygQRKQJLVy9jf96YTmlu6v4+sRB3DJ5KBmph9/Bf/30wWSmJfPDFwu44ek8Zl6TQ2aaQiEWFAgictR2VVRzb24hL3ywhSE9OvDwt07jxH5dol7/2lMGkJGSzB0v5HPdk+/xxPRxdEjX7qml6RsXkaMyv2Ard/99BTv3VXPzWcdz01nHH1G3z2Xj+pGemsSts5Zx9WPv8vRXx9O5XWozVCyHokCQVsHdueXPS1mwchu9O2dwbJdMju2SwbGdMzm2Sya9u2TQp0smvTpnqA+6hezYu58fzS4gN7+EEb078eT0cYzs0/motnnRiX3ISE3m5j8u4YpH3+HZ68fTrUN6E1UsDVEgSKvwtyVb+PvSYiYN70FyklG8q4qC4nK2763+TNvuHdM59pPQCF4R893ap+m696Pg7uTml/Cj2QXsqarhtslD+cYZg5vsCqEp2b149LocZjyTx+Uz3+EPN0ygR6eMJtm2HJ61puHjOTk5rkdoJp7iXZVM+eUbDO/diT/NOPlT16tX1YQoKa+iZFclW3ZVUryriuJdlRSXV4b/3FVFZU3oU9tLS06id8TRxbFdMj4THO3Vf31Q2/ZU8YO/r2B+QSlj+nbmp5eO4YReHZvlvRat28H1Ty+mR8d0/nDjyfTpktks75MIzOx9d89psJ0CQeJZXZ1zzRPvsuSjXcz7z4n079auUeu7O+WVNQcJi/B0ya5Ktu6uoq7ef4POmamfObKIDI6eHdNJSaBr5t2dFz7Ywr25hVTWhLht8lCu//zAZv8OPvjoY6574j06ZaTyhxsmMCCrfbO+X1ulQJA24el/beRHswv43y+O4soJ/ZvlPWpDdZTu2R8cUUSERXklW4Lp8sqaT62TZNCrUwa9g4A4oWcHzsnuxZAeHdrcvXlKyiv5rxeWs3B1GTnHdeUnl45mcPcOLfb+K7aUc83j75KanMQfbpjAkJ7Nc0TSlikQpNVbX7aXab9+k5MHdePJ6eNiuqPdt7/2UwER7qL69xHHRzsrcIeBWe05J7snU7N7MaZvl1Z9rsLdeW7xZv53zkpq65zbp57AtacMiMktJj4s3cNVj71LqM555mvjj/rkdaJRIEirVhuq49LfLWLD9n28fMtEesb5ScVtu6t4ubCU+QVbWbRuB7V1Ts9O6ZwzohdTR/Zi/MBjWtVtGTbvrODOF/J5e+0OTh50DD+5ZDTHdYttd82G7fu46tF32Lu/lqe/Np6x/bvGtJ7WpEkDwcymAg8CycBj7v7jej/vDzwNdAna3Onuc81sMvBjIA2oBr7n7q8G67wG9AYqg82c4+7bDleHAiFxPLRwLT+bv5rfXDGWC8YcG+tyGqW8ooZXV5cyf0Upr324jaqaOjpnpnL28B5Mze7FxKHdGxy9Gyt1dc6z72ziJ/NWkWTGXdOGccW4/nFzpFP0cQVXPfYu2/fs5/Hp4zh5ULdYl9QqNFkgmFky8CEwGSgCFgNXuHthRJuZwBJ3f9jMRgBz3X2AmY0FSt292MxGAvPdvU+wzmvAd9096j28AiExFBSXc/FDbzMluxe/vfJzsS7nqFRWh3hjTRnzC7ayoLCU3VW1ZKYmc/rQ7kwd2Yszh/Wgc2Z8DL7asH0fdzyfz3sbdzJxaHce+NKouLyyp3R3FVc99i5FH1fwyDU5nD60e6xLinvRBkI019aNB9a6+/pgw88BFwGFEW0c6BRMdwaKAdx9SUSbAiDDzNLdfX8U7ysJaH9tiFv/vIwu7dK476KRsS7nqGWmJTMluxdTsntRE6rj3fU7mV+wlZcLtzKvYCupycYpg7OYkt2TySN60qNjy3eNheqcJ97awM9fXk1aShI/u3Q0l57UN25PjvfslMGfZ5zM1Y+/x41P5/HbK8dyTnavWJfVJkRzhHApMNXdbwjmrwEmuPtNEW16Ay8DXYH2wCR3f/8g2/mGu08K5l8DugEh4K/A/3gDxegIoe174KWVPPL6ep6cPo4zh/WIdTnNpq7OWVq0i/kFW5m/Yisbd1RgBif17/pJgDT2EtsjsaZ0D997Pp+lm3cxaXgP7v/iqLg/X3NAeUUN1z75Hiu2lPOrr5zY6roWW1JTdhl9GZhSLxDGu/vNEW1uDbb1CzM7BXgcGOnudcHPs4HZhM8TrAuW9XH3LWbWkXAg/N7dnznI+88AZgD079//pE2bNkXx8aU1WrxxJ5c9sojLx/XjgS+NjnU5Lcbd+bB0L/MLtjJvxVYKS3YDMLx3J6Zk92RKdi+G9erYpL+x14TqmPnGeh5csIb26cncc2E2F445Nm6PCg5lT1UN1z+VR96mnfzkktF8OadfrEuKS00ZCKcA97j7lGD+LgB3fyCiTQHho4jNwfx64GR332ZmfYFXga+6+9uHeI/pQE7kUcfB6Aih7dq3v5ZzH3wTx3npPycm9J0uN++sCB85FGwlb9PHuMNx3doFRw49Gduv61Gd5C0s3s3tf13Gii27OW9Ub+65MJvuHVvv/YIqq0PMeDaPN9ds576LsrnmlAGxLinuNOU5hMXAEDMbCGwBLgeurNfmI+Bs4CkzGw5kAGVm1gWYA9wVGQZmlgJ0cfftZpYKnA8siKIWaaP+d+5KNn9cwZ9nnJLQYQDQ75h23PCFQdzwhUGU7dnPgpXhy1mffHsDM99YT4+O6UweET5yOHlQN9JSoructbq2jocWruWhhWvp0i6Vh6/6HOeO6t3Mn6b5ZaYl8+i1Odz0xw/4wYsFVNaEmDFxcKzLapWivex0GvArwpeUPuHu95vZvUCeu88Orix6FOhA+ATz7e7+spndDdwFrInY3DnAPuANIDXY5gLgVnf/9E1n6tERQtv02uptTH9yMTMmDuK/pg2PdTlxa3dVDQtXbWN+wVZeW11GRXWIThkpnD28J1OyezJxaHfapR08TJcXlfO955exauseLj7xWH50QTZd26e18CdoXjWhOr7z56XMyS/hlklD+Y+zj291XWDNRQPTpFXYVVHNlF+9QefMVGbf9Pm4vT4/3lTVhHhrzXbmFWxlwcpSdlXUkJGaxMQh3ZmS3YtJw3vSuV0qVTUhHnxlDTPfWE9WhzTuv3gUk0b0jHX5zSZU59z+fD5//aCIr58+iDunDlMo0LRdRiLN5ocvFrBjbzWPXzdOYdAIGanJTBrRk0kjelIbquO9jTuZv2IrLxeW8nJhKSlJxsmDulFSXsm6sn18Jacf/3Xe8LgZ89BckpOMn106msy0JB55fT1V1SF+dEF23Aysi3cKBImZ3PxiZi8r5tbJQ3VvmqOQkpzEqYOzOHVwFvdcmE1+UTnzgpPSODzztfFMTKDBW0lJxn0XjaRdWgoz31hPZU2IB740Oib3YGptFAgSE9t2V3H331cwpl8XvnWGTgA2FTNjTL8ujOnXhTumDot1OTFjZtx17jAyU5N58JU1VNXU8YvLxrSq+0nFggJBWpy7c+cLy6msDvGLL49JqOcKSMsxM26ZPJTMtGR+/NIqqmpC/ObKsXrE6mHof6K0uD8v3syrq7Zx57nDOL5Hy91XXxLTN04fzH9fmM3LhaXc+Mz7VFYf9mLGhKZAkBa1eWcF9+UWcsqgblynAUTSQq47dQA/vWQ0b64pY/qT77F3f22sS4pLCgRpMXV1zm1/WUaSGT+/bIyu/JAWddm4fvzqKyeSt+ljrnn83c88BU8UCNKCnnh7A+9t2MkPLxgRl7dVlrbvohP78NCVn2PFlnKufPQddu6rjnVJcUWBIC3iw9I9/HT+aiaP6MmlJ/WNdTmSwKaO7MWj1+awdttevvLIIrbtrop1SXFDgSDNriZUx62zltIhPYUHvjRKI0cl5s44oQdPfXU8W3ZVctkji9iyq7LhlRKALjuVZvebV9eyYstufnf1SWR1aL131ZS25ZTB3Xj2+glMf/I9LvvdIv5444Rme250qM6prAlRUV1LZXWIiuBVWR1i36eWBdM1oWBZ7SftZl6b0+yD6xQI0qyWbd7FQwvX8qWxfZg6Uk+1kvhy0nFd+dONJ3PN4+/y5d8t4onp4+jRMf3fO+ya2k/tvCsidtLhZcF0xA48cod/oO3+2rpG1ZWabGSmJtM+PYXMtGTapSVTE6ojOal5x1Do5nbSbKpqQpz36zepqA4x7zsT2/x9dKT1+rB0D1c99i5le6J/um9actInO+sDf7ZLTaFderAsNSW8LOLnmWkptEtNpn16MJ2WTGZqsG7av3f+TT2iWje3k5j76bzVrCvbx++vn6AwkLg2tGdH/vatU5lfUEp6SlLEjvzTO+1PfmNPTW6TI+wVCNIs/rVuO0+8vYHrTjmOzw/JinU5Ig3q27Ud139+YKzLiKm2F3ESc3uqavjeX/IZmNWeO8/VA29EWgsdIUiTu/cfhZSUV/L8N08lM003EhNpLXSEIE3qn4Wl/OX9Ir55xmA+179rrMsRkUZQIEiT2bF3P3e9kM/w3p34z7OHxrocEWkkdRlJk3B3vv+3FeyurOXZ68eQlqLfNURaG/2vlSbx96VbmFewlVsmD2V4706xLkdEjoACQY5aSXklP3yxgJOO68qMiYNiXY6IHCEFghwVd+f25/OpDTm/+PIYPchcpBVTIMhR+f07m3hzzXa+f95wBmQ1z43BRKRlKBDkiG3Yvo/7565k4tDuXDWhf6zLEZGjpECQI1IbquO2WUtJS07ip5eM1jMORNoAXXYqR+SRN9bzwUe7ePDyE+nVOSPW5YhIE9ARgjRaYfFufrXgQ84b1ZsLxxwb63JEpIkoEKRR9teGuHXWUjpnpnHfxSPVVSTShqjLSBrlVwvWsGrrHh6/Lodj2qfFuhwRaUJRHSGY2VQzW21ma83szoP8vL+ZLTSzJWaWb2bTguWTzex9M1se/HlWxDonBcvXmtmvTb9qxr33N+3kkdfX8ZWcfpw9vGesyxGRJtZgIJhZMvAQcC4wArjCzEbUa3Y3MMvdxwKXA/8vWL4duMDdRwHXAc9GrPMwMAMYErymHsXnkGa2b38tt85axrFdMrn7fD3jQKQtiuYIYTyw1t3Xu3s18BxwUb02Dhy4gU1noBjA3Ze4e3GwvADIMLN0M+sNdHL3RR5+qPMzwMVH+VmkGT3w0ko+2lnBz788ho4ZehymSFsUzTmEPsDmiPkiYEK9NvcAL5vZzUB7YNJBtnMJsMTd95tZn2A7kdvsE23R0rJe/7CM37/zETd8fiAnD+oW63JEpJlEc4RwsL59rzd/BfCUu/cFpgHPmtkn2zazbOAnwNcbsc0D684wszwzyysrK4uiXGlK5RU13P78Mo7v0YHvTjkh1uWISDOKJhCKgH4R830JuoQiXA/MAnD3RUAGkAVgZn2BvwHXuvu6iG32bWCbBNub6e457p7TvXv3KMqVpvSj2SvYsbeaX152IhmpehymSFsWTSAsBoaY2UAzSyN80nh2vTYfAWcDmNlwwoFQZmZdgDnAXe7+9oHG7l4C7DGzk4Ori64FXjzqTyNNau7yEv6+tJibzjqeUX07x7ocEWlmDQaCu9cCNwHzgZWEryYqMLN7zezCoNltwI1mtgz4EzA9OFl8E3A88AMzWxq8egTrfBN4DFgLrANeasoPJkdn254qvv+35Yzu25lvn3l8rMsRkRZg4f1265CTk+N5eXmxLqPNc3dueDqPN9duZ+5/fJ7je3SMdUkichTM7H13z2monW5dIZ/i7jz59kZeWbWNO6YOUxiIJBDdukI+sbyonPvmFPLehp18YUgWXz11QKxLEpEWpEAQSsor+dn81bzwwRa6tU/j/i+O5Cs5/UjS4zBFEooCIYHt21/LI2+sZ+Yb66hz+OYZg/nWGYM1ElkkQSkQElCozvnrB0X8fP5qtu3ZzwVjjuX2KSfQ75h2sS5NRGJIgZBg/rV2O/8zZyWFJbsZ278LD199Eicd1zXWZYlIHFAgJIh1ZXt5YO4qFqwspU+XTH5zxVjOH91bD7gRkU8oENq4j/dV8+Ara/j9O5vISE3mjqnD+OppA3QbChH5DAVCG1VdW8czizby61fWsHd/LVeM788tk4eS1SE91qWJSJxSILQx7s78glJ+/NJKNu6oYOLQ7nx/2nBO6KUBZiJyeAqENiRyYNmQHh146qvjOOOEHg2vKCKCAqFNqD+w7H8uHsnl4/qRkqw7k4hI9BQIrVhFdS2/ez0YWFYH3zh9MN86czCdNLBMRI6AAqEVqgsGlv0sGFh2/uje3DF1mAaWichRUSC0Mv9at53756ykoHg3J/bTwDIRaToKhFZifdleHnhpFf8sDA8s+/UVY7lAA8tEpAkpEOLcrorwwLJnF4UHlt0+9QS+dtpADSwTkSanQIhT1bV1PPvOJn79yhr2VNVw+fj+3DJpKN07amCZiDQPBUKccXdeLizlgbnhgWVfGJLF3eeN0MAyEWl2CoQ4smJLOfflFvKuBpaJSAwoEOLA1vKq8MCyJUUc004Dy0QkNhQIMVRRXcsjr69n5hvrCdU5X5+ogWUiEjsKhBia8cz7vLV2uwaWiUhcUCDEyJZdlby1djvfmTSE70waGutyRERQJ3WMzMkvBuBLY/vGuBIRkTAFQozk5pcwum9n+ndTN5GIxAcFQgx8tKOC/KJyzh/dO9aliIh8QoEQA7nLw91F00YpEEQkfigQYiB3WQlj+3ehb1d1F4lI/FAgtLD1ZXspLNnN+aOPjXUpIiKfElUgmNlUM1ttZmvN7M6D/Ly/mS00syVmlm9m04Ll3YLle83st/XWeS3Y5tLglRD3aJiTXwLAeeouEpE40+A4BDNLBh4CJgNFwGIzm+3uhRHN7gZmufvDZjYCmAsMAKqAHwAjg1d9V7l73tF9hNYlN7+EcQO60qtzRqxLERH5lGiOEMYDa919vbtXA88BF9Vr40CnYLozUAzg7vvc/S3CwZDw1pTuYXXpHnUXiUhciiYQ+gCbI+aLgmWR7gGuNrMiwkcHN0f5/k8G3UU/sAR49Nc/8ktIMjh3VK9YlyIi8hnRBMLBdtReb/4K4Cl37wtMA541s4a2fZW7jwK+ELyuOeibm80wszwzyysrK4ui3Pjk7szJL2bCwG706KjuIhGJP9EEQhHQL2K+L0GXUITrgVkA7r4IyACyDrdRd98S/LkH+CPhrqmDtZvp7jnuntO9e/coyo1Pq7buYV3ZPs7TYDQRiVPRBMJiYIiZDTSzNOByYHa9Nh8BZwOY2XDCgXDIX+fNLMXMsoLpVOB8YEXjy289cvOLSU4yzh2p7iIRiU8NXmXk7rVmdhMwH0gGnnD3AjO7F8hz99nAbcCjZnYL4e6k6e7uAGa2kfAJ5zQzuxg4B9gEzA/CIBlYADza5J8uTrg7ufklnDq4G9066JnIIhKforr9tbvPJXyyOHLZDyOmC4HTDrHugENs9qToSmz9VmzZzaYdFXzrjMGxLkVE5JA0UrkF5C4vJiXJmJKt7iIRiV8KhGYWvrqohM8PyaJLu7RYlyMickgKhGa2dPMuij6u1GA0EYl7CoRmlptfQlpyEudk94x1KSIih6VAaEZ1dc7c5SVMHNqdThmpsS5HROSwFAjN6IOPPqakvEpPRhORVkGB0Ixy80tIT0li0gh1F4lI/FMgNJNQnTNneQlnntCDDulRDfcQEYkpBUIzWbxxJ2V79nP+GHUXiUjroEBoJrn5xWSmJnPWsIR4EJyItAEKhGZQG6rjpeVbOWt4D9qlqbtIRFoHBUIzeGf9Tnbsq+YCXV0kIq2IAqEZzFleTPu0ZM44Qd1FItJ6KBCaWE2ojpdWbGXyiJ5kpCbHuhwRkagpEJrY22u3s6uihvN07yIRaWUUCE0sN7+EjhkpTBx62CeIiojEHQVCE9pfG2J+wVbOGdGL9BR1F4lI66JAaEJvrdnOnqpaDUYTkVZJgdCEcvNL6JyZymmD1V0kIq2PAqGJVNWE+GdhKVOze5GWoq9VRFof7bmayGury9i7X91FItJ6KRCayJzlJRzTPo1TBnWLdSkiIkdEgdAEKqtDvLKylKkje5GSrK9URFon7b2awKurtlFRHdKT0USkVVMgNIHc/GK6d0xnwkB1F4lI66VAOEr79tfy6qptTBvZi+Qki3U5IiJHTIFwlBasLGV/bR3nj9G9i0SkdVMgHKXc/BJ6dcrgpP5dY12KiMhRUSAchd1VNby+uoxpo3qTpO4iEWnlFAhHYUFhKdWhOg1GE5E2IapAMLOpZrbazNaa2Z0H+Xl/M1toZkvMLN/MpgXLuwXL95rZb+utc5KZLQ+2+Wsza3W/Yufml9CnSyZj+3WJdSkiIketwUAws2TgIeBcYARwhZmNqNfsbmCWu48FLgf+X7C8CvgB8N2DbPphYAYwJHhNPZIPECvlFTW8uaaM80b3phVmmYjIZ0RzhDAeWOvu6929GngOuKheGwc6BdOdgWIAd9/n7m8RDoZPmFlvoJO7L3J3B54BLj7yj9Hy5hdspSbkGowmIm1GNIHQB9gcMV8ULIt0D3C1mRUBc4Gbo9hmUQPbBMDMZphZnpnllZWVRVFuy/hHfjH9j2nHqD6dY12KiEiTiCYQDtYf4vXmrwCecve+wDTgWTM73Laj2WZ4oftMd89x95zu3btHUW7z27mvmn+t28H56i4SkTYkmkAoAvpFzPcl6BKKcD0wC8DdFwEZwOGeElMUbOdw24xb81ZsJVTnnKfuIhFpQ6IJhMXAEDMbaGZphE8az67X5iPgbAAzG044EA7Zv+PuJcAeMzs5uLroWuDFI6g/JnLzixmU1Z4RvTs13FhEpJVoMBDcvRa4CZgPrCR8NVGBmd1rZhcGzW4DbjSzZcCfgOnByWLMbCPwf8B0MyuKuELpm8BjwFpgHfBS032s5lO2Zz/vrFd3kYi0PSnRNHL3uYRPFkcu+2HEdCFw2iHWHXCI5XnAyGgLjRfzVpRQ5+jeRSLS5mikciP9I7+EIT06MLRnx1iXIiLSpBQIjVC6u4rFG3dy/mgdHYhI26NAaIQ5+SW4o3sXiUibpEBohDnLSxjeuxODu3eIdSkiIk1OgRCl4l2VvL/pY92qQkTaLAVClObklwAoEESkzVIgRCk3v5hRfTpzXLf2sS5FRKRZKBCisHlnBcuKynV0ICJtmgIhCrlBd5HuXSQibZkCIQq5+cWc2K8Lfbu2i3UpIiLNRoHQgA3b91FQvFvdRSLS5ikQGpC7LHxXbnUXiUhbp0BowJzlJYwb0JXenTNjXYqISLNSIBzG2m17WLV1D+eN0tGBiLR9CoTD+MeyEsxgmgJBRBKAAuEQ3J3c/GImDDyGHp0yYl2OiEizUyAcwurSPawr26dbXYtIwlAgHELushKSDM4d2SvWpYiItAgFwkEc6C46dXAW3Tqkx7ocEZEWoUA4iILi3WzcUaHBaCKSUBQIB5GbX0JKkjFV3UUikkAUCPUc6C76/JAsurRLi3U5IiItRoFQz7Kicoo+rtRgNBFJOAqEenKXFZOWnMQ52eouEpHEokCIUFfnzF1ewsShWXTOTI11OSIiLUqBEGHJ5o8pLq/SYDQRSUgKhAj/WFZCWkoSZw/vEetSRERanAIhEAq6i848oTsdM9RdJCKJR4EQWLxxJ9v27Fd3kYgkLAVCYE5+CZmpyeouEpGEFVUgmNlUM1ttZmvN7M6D/Ly/mS00syVmlm9m0yJ+dlew3mozmxKxfKOZLTezpWaW1zQf58jUhup4aUUJZw3vQbu0lFiWIiISMw3u/cwsGXgImAwUAYvNbLa7F0Y0uxuY5e4Pm9kIYC4wIJi+HMgGjgUWmNlQdw8F653p7tub8PMckXc37GT73mrO12A0EUlg0RwhjAfWuvt6d68GngMuqtfGgU7BdGegOJi+CHjO3fe7+wZgbbC9uJKbX0z7tGTOHKbuIhFJXNEEQh9gc8R8UbAs0j3A1WZWRPjo4OYo1nXgZTN738xmHOrNzWyGmeWZWV5ZWVkU5TZOTaiOeSu2MmlETzJSk5t8+yIirUU0gWAHWeb15q8AnnL3vsA04FkzS2pg3dPc/XPAucC3zWziwd7c3We6e46753Tv3j2KchvnX+t28HFFja4uEpGEF00gFAH9Iub78u8uoQOuB2YBuPsiIAPIOty67n7gz23A34hRV1LusmI6pqcwcWhWLN5eRCRuRBMIi4EhZjbQzNIInySeXa/NR8DZAGY2nHAglAXtLjezdDMbCAwB3jOz9mbWMWjfHjgHWNEUH6gxqmvrmF+wlcnZPUlPUXeRiCS2Bq8ycvdaM7sJmA8kA0+4e4GZ3Qvkufts4DbgUTO7hXCX0HR3d6DAzGYBhUAt8G13D5mZ/P5NAAAGVUlEQVRZT+BvZnaghj+6+7zm+ICH89baMnZX1XKBuotERBoOBAB3n0v4ZHHksh9GTBcCpx1i3fuB++stWw+MaWyxTS13WQmdM1M57Xh1F4mIJOxI5aqaEC8XljIluydpKQn7NYiIfCJh94Svf1jG3v21urpIRCSQsIEwJ7+EY9qncergbrEuRUQkLiRkIFRWh1iwspSpI3uRkpyQX4GIyGck5N5w4eptVFSHOH+07l0kInJAQgZCbn4xWR3SmTBQ3UUiIgckXCDs21/Lq6u2MW1UL5KTDnZnDRGRxJRwgfDKqm1U1dTp6iIRkXoSLhBylxXTs1M6Ocd1jXUpIiJxJaECYU9VDa99WMa0Ub1JUneRiMinJFQg/LOwlOpadReJiBxMQgXCnPwS+nTJ5HP9u8S6FBGRuJMwgVBeUcMba8o4b3RvgrusiohIhIQJhPmFW6kJOeeN0mA0EZGDSZhAyM0vof8x7Rjdt3OsSxERiUsJEQgf76vm7bXb1V0kInIYCREI8wq2Eqpz3btIROQwEiIQcvOLGZjVnhG9O8W6FBGRuBXVIzRbM3dneK9OnDWsp7qLREQOo80Hgplx9/kjYl2GiEjcS4guIxERaZgCQUREAAWCiIgEFAgiIgIoEEREJKBAEBERQIEgIiIBBYKIiABg7h7rGqJmZmXApiNcPQvY3oTlNBXV1Tiqq3FUV+O01bqOc/fuDTVqVYFwNMwsz91zYl1HfaqrcVRX46iuxkn0utRlJCIigAJBREQCiRQIM2NdwCGorsZRXY2juhonoetKmHMIIiJyeIl0hCAiIofR5gLBzKaa2WozW2tmdx7k5xPN7AMzqzWzS+OorlvNrNDM8s3sFTM7Lk7q+oaZLTezpWb2lpm1yMMlGqorot2lZuZm1iJXhkTxfU03s7Lg+1pqZjfEQ11Bm8uCf2MFZvbHeKjLzH4Z8V19aGa74qSu/ma20MyWBP8np8VJXccF+4d8M3vNzPo2aQHu3mZeQDKwDhgEpAHLgBH12gwARgPPAJfGUV1nAu2C6W8Cf46TujpFTF8IzIuHuoJ2HYE3gHeAnHioC5gO/LYl/l01sq4hwBKgazDfIx7qqtf+ZuCJeKiLcJ/9N4PpEcDGOKnrL8B1wfRZwLNNWUNbO0IYD6x19/XuXg08B1wU2cDdN7p7PlAXZ3UtdPeKYPYdoGmT/8jr2h0x2x5oiZNODdYVuA/4KVDVAjU1pq6WFk1dNwIPufvHAO6+LU7qinQF8Kc4qcuBAw9h7wwUx0ldI4BXgumFB/n5UWlrgdAH2BwxXxQsi7XG1nU98FKzVhQWVV1m9m0zW0d45/sf8VCXmY0F+rl7bgvUE3VdgUuCQ/rnzaxfnNQ1FBhqZm+b2TtmNjVO6gLCXSHAQODVOKnrHuBqMysC5hI+eomHupYBlwTTXwQ6mlm3piqgrQWCHWRZPFxGFXVdZnY1kAP8rFkrCt7uIMs+U5e7P+Tug4E7gLubvaoG6jKzJOCXwG0tUEukaL6vfwAD3H00sAB4utmriq6uFMLdRmcQ/k38MTPrEgd1HXA58Ly7h5qxngOiqesK4Cl37wtMA54N/t3Fuq7vAqeb2RLgdGALUNtUBbS1QCgCIn8j60vLHOo1JKq6zGwS8H3gQnffHy91RXgOuLhZKwprqK6OwEjgNTPbCJwMzG6BE8sNfl/uviPi7+5R4KRmrimquoI2L7p7jbtvAFYTDohY13XA5bRMdxFEV9f1wCwAd18EZBC+n1BM63L3Ynf/kruPJbyvwN3Lm6yC5j5R0pIvwr8FrSd86HngpEz2Ido+RcudVG6wLmAs4RNKQ+Lp+4qsB7gAyIuHuuq1f42WOakczffVO2L6i8A7cVLXVODpYDqLcNdEt1jXFbQ7AdhIMC4qTr6vl4DpwfRwwjvmZq0vyrqygKRg+n7g3iatoSX+AlryRfjw7sNg5/r9YNm9hH/rBhhHOIn3ATuAgjipawFQCiwNXrPjpK4HgYKgpoWH2zG3ZF312rZIIET5fT0QfF/Lgu9rWJzUZcD/AYXAcuDyeKgrmL8H+HFL1NOI72sE8Hbw97gUOCdO6roUWBO0eQxIb8r310hlEREB2t45BBEROUIKBBERARQIIiISUCCIiAigQBARkYACQUREAAWCiIgEFAgiIgLA/wd5bGPyu87ZpwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot CV误差曲线\n",
    "test_means = grid_search.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = grid_search.cv_results_[ 'std_test_score' ]\n",
    "train_means = grid_search.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = grid_search.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "x_axis = colsample_bytree_s\n",
    "\n",
    "plt.plot(x_axis, test_means)\n",
    "#plt.errorbar(x_axis, -test_scores[:,i], yerr=test_stds[:,i] ,label = str(max_depths[i]) +' Test')\n",
    "#plt.errorbar(x_axis, -train_scores[:,i], yerr=train_stds[:,i] ,label = str(max_depths[i]) +' Train')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 5 candidates, totalling 25 fits\n",
      "[CV] colsample_bytree=0.3 ............................................\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] colsample_bytree=0.3 ............................................\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8075559199130116, total=   4.4s\n",
      "[CV] colsample_bytree=0.3 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8354614694826217, total=   5.0s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8036485451323386, total=   5.3s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8309890813468244, total=   5.8s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.3, score=0.8181191437897483, total=   4.7s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.4, score=0.8216171039844509, total=   4.2s\n",
      "[CV] colsample_bytree=0.4 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .... colsample_bytree=0.4, score=0.848469101926485, total=   5.5s\n",
      "[CV] ... colsample_bytree=0.4, score=0.7991805387383246, total=   5.1s\n",
      "[CV] colsample_bytree=0.5 ............................................\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.4, score=0.8252768807712669, total=   5.2s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.4, score=0.8368928791330088, total=   5.9s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n",
      "[Parallel(n_jobs=4)]: Done  10 tasks      | elapsed:   19.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8145629680443606, total=   5.5s\n",
      "[CV] colsample_bytree=0.5 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8504405190647688, total=   6.7s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8049904013153456, total=   5.1s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .... colsample_bytree=0.5, score=0.834410238562153, total=   4.6s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.5, score=0.8241950013843464, total=   4.8s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.6, score=0.8475788029497513, total=   5.5s\n",
      "[CV] colsample_bytree=0.6 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.6, score=0.8089855370719716, total=   3.6s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ..... colsample_bytree=0.6, score=0.82667787153173, total=   4.8s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.6, score=0.8114303812375729, total=   6.1s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] .... colsample_bytree=0.6, score=0.826158684214272, total=   4.5s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.7, score=0.8557378379923398, total=   6.3s\n",
      "[CV] colsample_bytree=0.7 ............................................\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV] ... colsample_bytree=0.7, score=0.8117361230206368, total=   4.9s\n",
      "[CV] ... colsample_bytree=0.7, score=0.8064860581820926, total=   4.8s\n",
      "[CV] ... colsample_bytree=0.7, score=0.8294663080375202, total=   5.2s\n",
      "[CV] ... colsample_bytree=0.7, score=0.8334817162957552, total=   2.9s\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=4)]: Done  25 out of  25 | elapsed:   39.4s finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=StratifiedKFold(n_splits=5, random_state=3, shuffle=True),\n",
       "       error_score='raise',\n",
       "       estimator=LGBMClassifier(boosting_type='goss',\n",
       "        categorical_feature=[0, 1, 3, 5, 6, 12, 15, 16, 17, 18, 19, 20],\n",
       "        class_weight=None, colsample_bytree=1.0, importance_type='split',\n",
       "        is_unbalance=True, learning_rate=0.1, max_depth=6,\n",
       "        min_child_samples=40, min_child_weight=....0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0),\n",
       "       fit_params=None, iid=True, n_jobs=4,\n",
       "       param_grid={'colsample_bytree': [0.3, 0.4, 0.5, 0.6, 0.7]},\n",
       "       pre_dispatch='2*n_jobs', refit=False, return_train_score='warn',\n",
       "       scoring='roc_auc', verbose=5)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.1,\n",
    "          'n_estimators':n_estimators_1,\n",
    "          'num_leaves': 70,\n",
    "          'max_depth': 6,\n",
    "          #'colsample_bytree': 0.7,\n",
    "          'min_child_samples':40\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False, **params)\n",
    "#colsample_bytree_s = [i/10 for i in range(5,10)]\n",
    "#第一次使用5-10，第二次继续细调3-6\n",
    "colsample_bytree_s = [i/10 for i in range(3,8)]\n",
    "tuned_parameters = dict( colsample_bytree = colsample_bytree_s)\n",
    "grid_search = GridSearchCV(lg,n_jobs=4,param_grid=tuned_parameters,cv=kfold,scoring='roc_auc',verbose=5, refit = False)\n",
    "grid_search.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8273816607159732\n",
      "{'colsample_bytree': 0.7}\n"
     ]
    }
   ],
   "source": [
    "print(grid_search.best_score_)\n",
    "print(grid_search.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " 正则化参数lambda_l1(reg_alpha), lambda_l2(reg_lambda)感觉不用调了\n",
    " \n",
    " # 减小学习率进一步调试n_estimator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 423\n",
      "best cv score: 0.8441423408510296\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.01,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'num_leaves': 68,\n",
    "          'max_depth': 6,\n",
    "          'min_child_samples':40,\n",
    "          'colsample_bytree': 0.7\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , x_train , y_train, early_stopping_rounds=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best n_estimators: 147\n",
      "best cv score: 0.8325133003740757\n"
     ]
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.001,\n",
    "          #'n_estimators':n_estimators_1,\n",
    "          'num_leaves': 68,\n",
    "          'max_depth': 6,\n",
    "          'min_child_samples':40,\n",
    "          'colsample_bytree': 0.7\n",
    "         }\n",
    "n_estimators_2 = get_n_estimators(params , x_train , y_train, early_stopping_rounds=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 将最好的参数输入重新训练模型\n",
    "由于样本数目增多，模型复杂度稍微扩大一点？ num_leaves增多5\n",
    "#min_child_samples按样本比例增加到15"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/listen/anaconda3/lib/python3.7/site-packages/lightgbm/basic.py:741: UserWarning: categorical_feature keyword has been found in `params` and will be ignored.\n",
      "Please use categorical_feature argument of the Dataset constructor to pass this parameter.\n",
      "  .format(key))\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LGBMClassifier(boosting_type='goss',\n",
       "        categorical_feature=[0, 1, 3, 5, 6, 12, 15, 16, 17, 18, 19, 20],\n",
       "        class_weight=None, colsample_bytree=0.7, importance_type='split',\n",
       "        is_unbalance=True, learning_rate=0.001, max_depth=6,\n",
       "        min_child_samples=40, min_child_weight=0.001, min_split_gain=0.0,\n",
       "        n_estimators=423, n_jobs=4, num_leaves=73, objective='binary',\n",
       "        random_state=None, reg_alpha=0.0, reg_lambda=0.0, silent=False,\n",
       "        subsample=1.0, subsample_for_bin=200000, subsample_freq=0)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = {'boosting_type': 'goss',\n",
    "          'objective': 'binary',\n",
    "          'is_unbalance':True,\n",
    "          'categorical_feature': [0,1,3,5,6,12,15,16,17,18,19,20],\n",
    "          'n_jobs': 4,\n",
    "          'learning_rate': 0.001,\n",
    "          'n_estimators':423,\n",
    "          'num_leaves': 73,\n",
    "          'max_depth': 6,\n",
    "          'min_child_samples':40,\n",
    "          'colsample_bytree': 0.7\n",
    "         }\n",
    "lg = LGBMClassifier(silent=False, **params)\n",
    "lg.fit(x_train,y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 保存模型，用于后续测试¶"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "import _pickle\n",
    "_pickle.dump(lg,open(\"Happy_Customer_Bank.pkl\",\"wb\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "#特征重要性\n",
    "df = pd.DataFrame({\"columns\":list(feat_names),\"importance\":list(lg.feature_importances_.T)})\n",
    "df = df.sort_values(by=['importance'],ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>columns</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Monthly_Income</td>\n",
       "      <td>4359</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Interest_Rate</td>\n",
       "      <td>1987</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Loan_Amount_Applied</td>\n",
       "      <td>1894</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Existing_EMI</td>\n",
       "      <td>1851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Loan_Amount_Submitted</td>\n",
       "      <td>1728</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>Age</td>\n",
       "      <td>1558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>City</td>\n",
       "      <td>1393</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Processing_Fee</td>\n",
       "      <td>1276</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>EMI_Loan_Submitted</td>\n",
       "      <td>1254</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Employer_Name</td>\n",
       "      <td>995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Loan_Tenure_Applied</td>\n",
       "      <td>875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Salary_Account</td>\n",
       "      <td>871</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Loan_Tenure_Submitted</td>\n",
       "      <td>811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Source</td>\n",
       "      <td>498</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Device_Type</td>\n",
       "      <td>387</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Var5</td>\n",
       "      <td>385</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gender</td>\n",
       "      <td>241</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Var4</td>\n",
       "      <td>193</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Var1</td>\n",
       "      <td>192</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Filled_Form</td>\n",
       "      <td>118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Mobile_Verified</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Var2</td>\n",
       "      <td>17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                  columns  importance\n",
       "13         Monthly_Income        4359\n",
       "7           Interest_Rate        1987\n",
       "8     Loan_Amount_Applied        1894\n",
       "4            Existing_EMI        1851\n",
       "9   Loan_Amount_Submitted        1728\n",
       "21                    Age        1558\n",
       "0                    City        1393\n",
       "14         Processing_Fee        1276\n",
       "2      EMI_Loan_Submitted        1254\n",
       "3           Employer_Name         995\n",
       "10    Loan_Tenure_Applied         875\n",
       "15         Salary_Account         871\n",
       "11  Loan_Tenure_Submitted         811\n",
       "16                 Source         498\n",
       "1             Device_Type         387\n",
       "20                   Var5         385\n",
       "6                  Gender         241\n",
       "19                   Var4         193\n",
       "17                   Var1         192\n",
       "5             Filled_Form         118\n",
       "12        Mobile_Verified          71\n",
       "18                   Var2          17"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAADi5JREFUeJzt3X+o3fV9x/Hnq1G7sY6qNYokcdetYWj/mEpQwTGcDo06FgsNpIw2FEf2RzosFDbtP25tBfvHqiusQlaDaelqpe2mtIIL/qDbH1Vjdf4Kksw6zRSTErUdpY7oe3+cT+ppcm7uufHmnHvzeT4gnO/3/f2ccz/fLyf3dT6f7/d8b6oKSVJ/3jftDkiSpsMAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXqhGl34EhOO+20mpmZmXY3JGlJefzxx39aVcvnareoA2BmZoYdO3ZMuxuStKQk+e9x2jkFJEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnVrU3wSW9OtmbvjB2G1fvOWaY9gTHQ8cAUhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpU2MHQJJlSZ5I8v22fnaSR5LsSvLtJCe1+vvb+u62fWboNW5s9eeTXLnQOyNJGt98RgDXAzuH1r8E3FpVq4HXgeta/Trg9ar6MHBra0eSc4ENwEeAtcBXkyx7b92XJB2tsQIgyUrgGuBrbT3AZcB3WpNtwLVteV1bp22/vLVfB9xVVW9V1U+A3cCFC7ETkqT5G3cEcBvw18A7bf1DwBtVdaCt7wFWtOUVwMsAbfubrf2v6iOeI0masDkDIMmfAnur6vHh8oimNce2Iz1n+OdtSrIjyY59+/bN1T1J0lEaZwRwCfBnSV4E7mIw9XMbcHKSg39RbCXwSlveA6wCaNs/COwfro94zq9U1ZaqWlNVa5YvXz7vHZIkjWfOAKiqG6tqZVXNMDiJ+2BV/TnwEPCx1mwjcE9bvret07Y/WFXV6hvaVUJnA6uBRxdsTyRJ8/Je/ibw3wB3Jfki8ARwR6vfAXwjyW4Gn/w3AFTVs0nuBp4DDgCbq+rt9/DzJUnvwbwCoKoeBh5uyy8w4iqeqvolsH6W598M3DzfTkqSFp7fBJakThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTcwZAkt9I8miS/0zybJK/a/WzkzySZFeSbyc5qdXf39Z3t+0zQ691Y6s/n+TKY7VTkqS5jTMCeAu4rKr+ADgPWJvkYuBLwK1VtRp4Hbiutb8OeL2qPgzc2tqR5FxgA/ARYC3w1STLFnJnJEnjmzMAauB/2+qJ7V8BlwHfafVtwLVteV1bp22/PEla/a6qequqfgLsBi5ckL2QJM3bWOcAkixL8iSwF9gO/BfwRlUdaE32ACva8grgZYC2/U3gQ8P1Ec8Z/lmbkuxIsmPfvn3z3yNJ0ljGCoCqeruqzgNWMvjUfs6oZu0xs2ybrX7oz9pSVWuqas3y5cvH6Z4k6SjM6yqgqnoDeBi4GDg5yQlt00rglba8B1gF0LZ/ENg/XB/xHEnShI1zFdDyJCe35d8E/gTYCTwEfKw12wjc05bvbeu07Q9WVbX6hnaV0NnAauDRhdoRSdL8nDB3E84EtrUrdt4H3F1V30/yHHBXki8CTwB3tPZ3AN9IspvBJ/8NAFX1bJK7geeAA8Dmqnp7YXdHkjSuOQOgqp4Czh9Rf4ERV/FU1S+B9bO81s3AzfPvpiRpoflNYEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktQpA0CSOmUASFKnDABJ6pQBIEmdMgAkqVMGgCR1ygCQpE4ZAJLUqTkDIMmqJA8l2Znk2STXt/qpSbYn2dUeT2n1JPlKkt1JnkpywdBrbWztdyXZeOx2S5I0l3FGAAeAz1bVOcDFwOYk5wI3AA9U1WrggbYOcBWwuv3bBNwOg8AAbgIuAi4EbjoYGpKkyZszAKrq1ar6cVv+ObATWAGsA7a1ZtuAa9vyOuDrNfAj4OQkZwJXAturan9VvQ5sB9Yu6N5IksY2r3MASWaA84FHgDOq6lUYhARwemu2Anh56Gl7Wm22uiRpCk4Yt2GSDwDfBT5TVT9LMmvTEbU6Qv3Qn7OJwdQRZ5111rjd0yI1c8MPxm774i3XHMOeSDrUWCOAJCcy+OX/zar6Xiu/1qZ2aI97W30PsGro6SuBV45Q/zVVtaWq1lTVmuXLl89nXyRJ8zDOVUAB7gB2VtWXhzbdCxy8kmcjcM9Q/ZPtaqCLgTfbFNH9wBVJTmknf69oNUnSFIwzBXQJ8Ang6SRPttrngFuAu5NcB7wErG/b7gOuBnYDvwA+BVBV+5N8AXistft8Ve1fkL2QJM3bnAFQVf/B6Pl7gMtHtC9g8yyvtRXYOp8OSpKOjbFPAkuT5Mlj6djzVhCS1ClHAEvUfD4hg5+SJR3OEYAkdcoAkKROGQCS1CkDQJI6ZQBIUqe8CkjHFb8/II3PEYAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ3yewCSNCGL7XsqjgAkqVMGgCR1ygCQpE4ZAJLUKQNAkjplAEhSpwwASerUcf09gMV2za0kLSaOACSpUwaAJHXKAJCkThkAktSp4/oksKQBL4jQKAbAApnPfzDwP5mk6XMKSJI6ZQBIUqcMAEnqlAEgSZ0yACSpUwaAJHXKAJCkThkAktSpOQMgydYke5M8M1Q7Ncn2JLva4ymtniRfSbI7yVNJLhh6zsbWfleSjcdmdyRJ4xpnBHAnsPaQ2g3AA1W1GnigrQNcBaxu/zYBt8MgMICbgIuAC4GbDoaGJGk65gyAqvohsP+Q8jpgW1veBlw7VP96DfwIODnJmcCVwPaq2l9VrwPbOTxUJEkTdLTnAM6oqlcB2uPprb4CeHmo3Z5Wm61+mCSbkuxIsmPfvn1H2T1J0lwW+mZwGVGrI9QPL1ZtAbYArFmzZmSb44k3kZM0LUc7AnitTe3QHve2+h5g1VC7lcArR6hLkqbkaAPgXuDglTwbgXuG6p9sVwNdDLzZpojuB65Icko7+XtFq0mSpmTOKaAk3wIuBU5LsofB1Ty3AHcnuQ54CVjfmt8HXA3sBn4BfAqgqvYn+QLwWGv3+ao69MSyJGmC5gyAqvr4LJsuH9G2gM2zvM5WYOu8eidJOmb8i2AS/snE2Xhcjm/eCkKSOmUASFKnnAKS3gOnSLSUOQKQpE4ZAJLUKQNAkjplAEhSpwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI6ZQBIUqcMAEnqlAEgSZ3ydtCSFpy3yV4aHAFIUqcMAEnqlAEgSZ0yACSpU54EHsETWJJ6YABI0jwdLx8SnQKSpE4ZAJLUKQNAkjplAEhSpwwASeqUVwFpLMfLVQ+S3uUIQJI6ZQBIUqecApK0aEx6qrH3qU1HAJLUKQNAkjrlFFBn5jPkheNz2CtpwBGAJHVq4iOAJGuBfwCWAV+rqlsm3Qdp2no/+ajFYaIjgCTLgH8ErgLOBT6e5NxJ9kGSNDDpKaALgd1V9UJV/R9wF7Buwn2QJDH5KaAVwMtD63uAiybcB0nHGafUjk6qanI/LFkPXFlVf9HWPwFcWFV/NdRmE7Cprf4+8PwCd+M04KcL/JrHA4/LaB6X0Twuh1tMx+R3qmr5XI0mPQLYA6waWl8JvDLcoKq2AFuOVQeS7KiqNcfq9Zcqj8toHpfRPC6HW4rHZNLnAB4DVic5O8lJwAbg3gn3QZLEhEcAVXUgyaeB+xlcBrq1qp6dZB8kSQMT/x5AVd0H3DfpnzvkmE0vLXEel9E8LqN5XA635I7JRE8CS5IWD28FIUmd6ioAkqxN8nyS3UlumHZ/FoskLyZ5OsmTSXZMuz/TkmRrkr1JnhmqnZpke5Jd7fGUafZx0mY5Jn+b5H/a++XJJFdPs4/TkGRVkoeS7EzybJLrW31JvV+6CQBvQzGnP66q85baZWwL7E5g7SG1G4AHqmo18EBb78mdHH5MAG5t75fz2nm93hwAPltV5wAXA5vb75Ml9X7pJgDwNhSaQ1X9ENh/SHkdsK0tbwOunWinpmyWY9K9qnq1qn7cln8O7GRwp4Ml9X7pKQBG3YZixZT6stgU8G9JHm/fxNa7zqiqV2Hwnx44fcr9WSw+neSpNkW0qKc5jrUkM8D5wCMssfdLTwGQETUvgRq4pKouYDA9tjnJH027Q1rUbgd+DzgPeBX4++l2Z3qSfAD4LvCZqvrZtPszXz0FwJy3oehVVb3SHvcC/8JgukwDryU5E6A97p1yf6auql6rqrer6h3gn+j0/ZLkRAa//L9ZVd9r5SX1fukpALwNxQhJfivJbx9cBq4Anjnys7pyL7CxLW8E7pliXxaFg7/gmo/S4fslSYA7gJ1V9eWhTUvq/dLVF8Ha5Wq38e5tKG6ecpemLsnvMvjUD4Nvhv9zr8clybeASxnc1fE14CbgX4G7gbOAl4D1VdXNSdFZjsmlDKZ/CngR+MuD8969SPKHwL8DTwPvtPLnGJwHWDLvl64CQJL0rp6mgCRJQwwASeqUASBJnTIAJKlTBoAkdcoAkKROGQCS1CkDQJI69f9NzrmI6rVz5QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar(range(len(lg.feature_importances_)),lg.feature_importances_)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
